Eliminate Before Align: A Remote Sensing Image-Text Retrieval Framework with Keyword Explicit Reasoning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 10:58:07

Abstract: Mountains of researches center around the Remote Sensing Image-Text Retrieval (RSITR), aiming at retrieving the corresponding targets based on the given query. Among them, the transfer of Foundation Models (FMs), such as CLIP, to remote sensing domain shows promising results. However, existing FM-based approaches neglect the negative impact of weakly correlated sample pairs and the key distinctions among remote sensing texts, leading to biased and superficial exploration of sample pairs. To address these challenges, we propose a novel Eliminate Before Align strategy with Keyword Explicit Reasoning framework (EBAKER) for RSITR. Specifically, we devise an innovative Eliminate Before Align (EBA) strategy to filter out the weakly correlated sample pairs to mitigate their deviations from optimal embedding space during alignment. Moreover, we introduce a Keyword Explicit Reasoning (KER) module to facilitate the positive role of subtle key concept differences. Without bells and whistles, our method achieves a one-step transformation from FM to RSITR task, obviating the necessity for extra pretraining on remote sensing data. Extensive experiments on three popular benchmark datasets validate that our proposed EBAKER method outperform the state-of-the-art methods with fewer training data. Our source code will be released soon.

Main paper: 
 introduction: 
 With the advancement of aerospace technology, remote sensing imagery has become increasingly accessible and finds wide applications in disaster monitoring [17] , navigation [36] , and agricultural production [33] . Among these applications, Remote Sensing Image-Text Retrieval (RSITR) stands as a foundational technique in remote sensing vision language domain [19] , aiming to retrieve semantically similar images based on given text queries, and vice versa.Recent research efforts have shifted towards RSITR, with the design of several effective methods leveraging from convolutional Figure 1 : Illustration of weakly correlated pairs. An image labeled as "A large number of trees are planted on both sides of the road" for a "viaduct" is weakly correlated, whereas "a medium residential" is strongly correlated but considered a negative image-text pair.neural networks [40] [41] [42] to Foundation Model (FM) [21, 45] . In the realm of RSITR, initial research efforts primarily revolve around CNN-based approaches [24, 25] . Abdullah et al. [1] pioneered the exploration of the RSITR problem by employing an average fusion strategy to attain robust representations. After that, plenty of CNNbased approaches [20, 27, 34, 38, 41, 42, 42] focused on refining alignment tailored to the characteristics of RSITR task. In recent years, with the flourishing development of FMs [4, 7, 15, 16, 29, 31] and their outstanding performance in various downstream tasks of image-text retrieval [10, 43, 44] , such as text-based person search [35] and product search [3] , researches in RSITR have pivoted towards the transfer from FM to Remote Sensing Image-Text Retrieval Model (RSITRM) [14, 45] . Yuan et al. [37] explored multiple Parameter-Efficient Fine-Tuning strategies to transfer CLIP to the remote sensing domain. Liu et al. [21] annotated and amalgamated multiple remote sensing datasets and compared the performance of different large-scale models such as CLIP [29] , BLIP [16] , and AL-BEF [18] in the remote sensing domain. Zhang et al. [45] proposed a 5M remote sensing dataset and achieved excellent performance by employing a two-step approach involving RS pretraining and downstream task fine-tuning to adapt CLIP to the remote sensing domain.Regardless of whether they are traditional or FM-based approaches, they both necessitate more refined datasets. Finely and accurately annotated RSITR data will contribute to improving the performance of the model [45] . However, despite careful annotation, meaningless labeled image-text pairs still exist in the dataset [30] . Meaningless or weakly correlated positive image-text pairs may mislead the alignment of semantically relevant instances. For instance, as shown in Figure 1 , an image of a "viaduct" may be labeled as "a large number of trees are planted on both sides of the road", which may be more relevant to a "mediumresidential". Such description, however, offers no benefit to the model. Therefore, it is worth exploring how to enable the model to autonomously eliminate the negative impact of such noise before fine-grained alignment.In addition, existing FMs boost the development of RSITR on data volume [21, 45] , which fail to grasp the core issue. The key of RSITR task lies not merely in increasing the quantity of positive and Conference'17, July 2017, Washington, DC, USA Anonymous Authors (a) Traditional Methods (e.g. RemoteCLIP [21] , GeoRSCLIP [45] )A dry river is in a piece of desert .A dry river is in a piece of desert .A dry river is in a piece of desert .A dry river is in a piece of desert . A large piece of green ocean is near a yellow beach .A large piece of green ocean is near a yellow beach .A large piece of green ocean is near a yellow beach .A baseball field near four tennis courts and a playground are in a school with several buildings and some green trees .A baseball field near four tennis courts and a playground are in a school with several buildings and some green trees .A baseball field near four tennis courts and a playground are in a school with several buildings and some green trees .A dry river is in a piece of desert . A large piece of green ocean is near a yellow beach .A baseball field near four tennis courts and a playground are in a school with several buildings and some green trees . A dry river is in a piece of desert .A dry river is in a piece of desert .A dry river is in a piece of desert . A large number of trees are planted on both sides of the road .A large number of trees are planted on both sides of the road .A large piece of green ocean is near a yellow beach .A large piece of green ocean is near a yellow beach .A large piece of green ocean is near a yellow beach .A baseball field near four tennis courts and a playground are in a school with several buildings and some green trees .A baseball field near four tennis courts and a playground are in a school with several buildings and some green trees .A dry river is in a piece of desert . A large number of trees are planted on both sides of the road .A large piece of green ocean is near a yellow beach .A baseball field near four tennis courts and a playground are in a school with several buildings and some green trees . negative sample pairs in contrastive learning, but delving deeply into the key differences of sample pairs. Existing traditional and CLIP-based methods in RSITR predominantly utilize global features from vision and text encoders as their output, neglecting the key differences guided by the fine-grained features within remote sensing images.Eliminate Before AlignThe aforementioned two challenges have constrained the potential of FM, despite their significant advancements in various traditional multi-modal downstream tasks, as exemplified by CLIP [29] and BLIP [16] . The adaptation of FMs to RSITRMs still necessitates a substantial volume of remote sensing data, as illustrated in Figure 2 . For instance, GeoRSCLIP [45] employs an additional 5M remote sensing image-text pairs and a two-step training to transfer CLIP to the RSITRM. This undoubtedly imposes an additional burden on training and presents challenging to achieving cost-effective performance growth.To this end, we propose a novel framework called Eliminate Before Align strategy with Keyword Explicit Reasoning (EBAKER) for achieving a one-step transition from FM to RSITRM, as illustrated in Figure 2 (b) . Specifically, we introduce an innovative Eliminate Before Align (EBA) strategy to counteract the adverse effects of weakly correlated pairs. Additionally, a Keyword Explicit Reasoning (KER) module is introduced to facilitate the positive role of subtle key concept differences. We validate the efficacy of our proposed EBAKER method on three popular benchmark datasets, i.e., RSICD [23] , RSITMD [38] , and NWPU [5] . Extensive experiments demonstrate that EBAKER consistently outperforms state-of-theart approaches.Our contributions are summarized as follows:• To achieve a one-step training from FM to RSITRM, we propose a novel Eliminate Before Align strategy with Keyword Explicit Reasoning framework (EBAKER), which aims at conducting fine-grained alignment through in-depth analysis of subtle distinctions and noise filtration. Our approach diverges from current state-of-the-art method GeoRSCLIP [45] by relying on a mere 4% of the training data and addressing the nuanced characteristics. • To mitigate the negative impact of the weakly correlated pairs, we devise an innovative Eliminate Before Align (EBA) strategy. It enables autonomously eliminate the positive sample pairs with low global similarity before alignment, which promotes the accuracy of fine-grained contrastive learning and boosts the intrinsic confidence of the model. • We introduce a Keyword Explicit Reasoning (KER) module, encouraging the model to predict subtle distinctions in key concepts within local features of remote sensing image-text pairs. This module facilitates fine-grained contrastive learning, enhancing the differentiation between extremely similar sample pairs. 
method: 
 In this section, we present our proposed EBAKER framework, as illustrated in Figure 3 . We begin with vision encoder, text encoder, and our keyword statistics and mask generation in Section 2. EQUATIONwhere 𝜙 represents text encoder of CLIP.We initially perform a statistical analysis to identify key concepts that require masking. Through word frequency analysis across the entire dataset, we exclude common high-frequency words such as "a", "the", "of", etc. Subsequently, we select top-𝑘 frequency keywords in each dataset, yielding the corresponding keyword list. The process of keyword statistics can be summarized as follows:EQUATIONWe merge the keyword lists from each datset and remove any duplicate words across them, resulting in the final keyword list for training. If a word in the input text matches a word in the keyword list, it is replaced with "[𝑚𝑎𝑠𝑘]". Accordingly, we generate the masked text 𝑇 𝑚𝑎𝑠𝑘 . Subsequently, we input the sentences after EQUATIONwhere ∥•∥ 2,2 represents the consecutive application of the L2-norm twice. During the inference stage, we combine the global similarity and local similarity via a weighted approach to obtain the final similarity between image and text:EQUATIONNext, we introduce our proposed EBA strategy as shown in Figure 4 . Specifically, we begin with conducting several epochs of regular training without eliminating any training samples, allowing the model to encounter all training samples during this process. Subsequently, we establish the Similarity Bank (𝑆𝑖𝑚𝐵𝑎𝑛𝑘), wherein we record the global similarity scores of all sample pairs within the current epoch:EQUATIONwhere 𝑆𝑖𝑚 𝑔 𝑖 represents the 𝑖-th global similarity, 𝐿 is the total number of image-text pairs in the dataset. Upon completion of an epoch, we extract all similarity values and sort them in descending order. We then select the similarity from the end of the sorted list by the predetermined drop ratio of the data volume, which will be served as the threshold 𝑇ℎ for the next epoch:EQUATIONwhere 𝑆𝑜𝑟𝑡 indicates sorting in descending order and 𝑑𝑟𝑜𝑝 𝑟𝑎𝑡𝑖𝑜 represents the specified elimination rate. During the training of the next epoch, the similarity of all matched image-text pairs within each batch is compared with the threshold 𝑇ℎ. If the similarity of the current image-text sample pair does not exceed the threshold 𝑇ℎ, its loss is excluded from the current batch. Suppose there are 𝑀 instances within a batch that do not exceed the threshold. When calculating the loss for image-to-text and text-to-image, the corresponding rows are removed, transforming the 𝑁 × 𝑁 matrix into an (𝑁 -𝑀) × 𝑁 matrix before alignment:EQUATIONwhere 𝑆𝑖𝑚 𝑖 𝑔 represents the global similarities in the 𝑖-th row, 𝐵 𝑠 𝑔 represents the global similarity matrix within a batch. If a global similarity is eliminated, the corresponding local similarity will also be eliminated. Similarly, the formulation can be expressed as:EQUATION)Following [12] , we adopt a single cross-attention layer, along with four Transformer blocks and a final Masked Language Modeling (MLM) head, to construct the word reasoning architecture. However, we observe that employing implicit reasoning on random tokens might overlook the subtle yet crucial differences among remote sensing images, which will be deeply analyzed in Section 3.5. EQUATIONwhere 𝐶𝐴 represents cross attention layer, which reasons the relationship between Q, K, and V, 𝑇𝑟𝑎𝑛𝑠 𝑓 𝑜𝑟𝑚𝑒𝑟 𝑁 represents 𝑁 Transformer blocks in accordance with CLIP [29] , 𝑀𝐿𝑀 ℎ𝑒𝑎𝑑 represents an MLP structure composed of a linear layer, QuickGELU layer, LayerNorm layer, and another linear layer.𝑝 𝑚 𝑖 |𝑚 ∈ 𝐿𝑖𝑠𝑡 𝑘𝑒𝑦 𝑀 𝑖=1denotes the predicted probability 𝑝 at position 𝑖 for the mask 𝑚 of the 𝐿𝑖𝑠𝑡 𝑘𝑒𝑦 . The loss function is defined as follows:EQUATIONwhere 𝑀 represents the number of masked tokens, 𝑉 is the vocabulary size of CLIP, 𝑦 𝑚 𝑖 is the one-hot distribution of the 𝑚-th masked word in the vocabulary of CLIP corresponding to the 𝑖-th token.We employ the prevalant InfoNCE loss [26] to achieve fine-grained alignment, applied to both global and local similarities, which can be expressed as follows: EQUATIONEQUATIONwhere 𝑁 -𝑀 𝑖=1 𝑠 𝑣𝑡 𝑖 𝑗 and 𝑁 -𝑀 𝑖=1 𝑠 𝑡 𝑣 𝑖 𝑗 respectively represent the sum of each row in the similarity matrices for Image-to-Text or Text-to-Image after removing 𝑀 rows.Both global and local alignment utilize the InfoNCE loss, while the modeling of masked attributes employs the MLM loss. We set a drop epoch 𝐾, before which we employ the original InfoNCE loss to expose the model to all data. Once the epoch exceeds 𝐾, we switch to the modified InfoNCE loss to eliminate noise. The overall loss function is formulated as: Image-to-Text Retrieval Text-to-Image Retrieval mR R@1 R@5 R@10 R@1 R@5 R@10 VSE++ BMVC2018 [9] ResNet18/Bi-GRU L 𝑡𝑜𝑡𝑎𝑙 = L 𝑔 𝑖𝑛𝑓 𝑜 + L 𝑙 𝑖𝑛𝑓 𝑜 + 𝛼 L 𝑚𝑙𝑚 , 𝑖 𝑓 𝑒𝑝𝑜𝑐ℎ < 𝐾 L 𝑔 𝑖𝑛𝑓 𝑜 + L 𝑙 𝑖𝑛𝑓 𝑜 + 𝛼 L 𝑚𝑙𝑚 , 𝑖 𝑓 𝑒𝑝𝑜𝑐ℎ ≥ 𝐾 
experiments 3.1 datasets: 
 In our experiments, we employ three benchmark datasets, RSICD [23] , RSITMD [38] , and NWPU [5] , to validate the effectiveness of our approach. Following the methodology of RemoteCLIP [21] , we generate 𝑝-Hash values for image-text pairs and set a threshold of 2 to merge these three datasets into one, aiming to eliminate redundant images. The RSICD dataset comprises 10,921 images, each sized at 224×224 pixels, making it the most widely utilized dataset for RSITR. RSITMD consists of 4,743 images, each with a size of 256×256 pixels. The NWPU dataset encompasses 31,500 images, also with a size of 256×256 pixels. Following the protocol of [38] , we divide these three datasets into train sets (80%), validation sets (10%), and test sets (10%).We utilize Recall at 𝑘 (𝑅@𝑘, 𝑘=1,5,10) and mean Recall (mR) as the evaluation metrics to evaluate the retrieval performance. Particularly, 𝑅@𝑘 measures the percentage of ground truth instances within the top 𝑘 samples, while mR represents the average value across all six 𝑅@𝑘 metrics, providing an overall assessment of retrieval performance.The implementation of EBAKER is derived from the RemoteCLIP codebase [21] , with the ViT-B-32 architexture provided by Open-CLIP [6] . We train the model for 7 epochs with a batch size of 100 and employ Adam [13] as our optimizer. Additionally, we adopt linear warm-up and cosine learning rate scheduler. The learning rate is set to 1.5e-5 and weight decay is set to 0. We compare our proposed EBAKER with existing state-of-the-art methods, which include traditional methods like VSE++ [9] , LW-MCR [39] , AMFMN [38] , GaLR [41] , KCR [25] , Multilanguage [2] , SWAN [28] , FAMMI [46] , PIR [27] , and KAMCL [11] . Additionally, we cover CLIP-based methods such as PE-RSITR [37] , RemoteCLIP [21] , and GeoRSCLIP [45] . Our preposed EBAKER falls under the category of CLIP-based methods. The experimental results for the RSICD [23] , RSITMD [38] , and NWPU [5] datasets are presented in Table 1 . Additionally, the vision and text backbones (separated by "/"), total train set, and test set are provided within the table. From these results, we draw the following observations and conclusions.Comparison on RSICD, RSITMD, and NWPU. For the RSICD dataset, our EBAKER method notably outperforms all competing methods across various evaluation metrics. Specifically, compared with GeoRSCLIP [45] , our EBAKER method surpasses it on all evaluation metrics, while utilizing only 0.2 million data samples, a mere 4% of the 5.07 million utilized by GeoRSCLIP. Particularly noteworthy is the 3.29% improvement in image-to-text R@10, a 1.78% enhancement in text-to-image R@1, and an overall mR increase of 1.83%. For the RSITMD dataset, also compared with GeoRSCLIP, we achieve a 1.77% improvement in image-to-text R@1, a 3.01% enhancement in text-to-image R@1, and an overall mR increase of 1.51%. These results demonstrate a comprehensive performance superiority over GeoRSCLIP. Following [11] , we also conduct comparative experiments on the NWPU dataset, where the results of RemoteCLIP [21] are reproduced by fine-tuning on NWPU by the code and weight files provided in [21] . The results indicate that our method achieves superior performance with less data. Specifically, we observe a 3.49% improvement in image-to-text R@10, a 1.02% enhancement in text-to-image R@10, and an overall mR increase of 1.46%.As indicated by the backbones specified in Table 1 , we categorize the methods into traditional approaches and CLIP-based methods. Compared with traditional approaches, CLIP-based methods often achieve better performance through fine-tuning. However, they typically require more training data. For instance, GeoRSCLIP [45] necessitates an additional 5M remote sensing dataset for the process of RS pretraining, as shown in Figure 2 (a) . Our proposed EBAKER method achieves a balance between performance and computational cost by solely relying on the combination of the RSICD, RSITMD, and NWPU datasets, without the need for remote sensing data for an extra RS pretraining process. Compared with the traditional method KAMCL [11] , we achieve performance improvements of 14.60% and 17.18% in mR on the RSICD and RSITMD datasets, respectively. In comparison to the CLIP-based method GeoRSCLIP [45] , we only need a one-step fine-grained training, thereby eliminating the need for additional pretraining samples, and achieving performance improvements of 1.83% and 1.51% in mR, respectively.Image-to-Text Retrieval Image-to-Text Retrieval mR R@1 R@5 R@10 R@1 R@5 R@10 Approach Image-to-Text Retrieval Image-to-Text Retrieval mR R@1 R@5 R@10 R@1 R@5 R@10 IRR [12] 20 In this section, we design a variety of ablation experiments, which aim at investigating the performance gains of different modules in the model to verify the effectiveness of each part.As shown in Table 2 , we conduct ablation studies on different modules to demonstrate the effectiveness of our method. Initially, we choose the original CLIP as the baseline, and incorporate local alignment (local), the KER module, and the EBA strategy, respectively. Compared with the baseline, these additions resulted in respective improvements of 1.43%, 1.41%, and 1.42% in terms of mR. Subsequently, we conduct the combinations of each two modules and find that local alignment with KER module yields promising results, with an improvement of approximately 1.10% compared with utilizing either mechanism individually. This may be attributed to the fact that the reasoning ability of key concepts of KER explicitly manifests in the fine-grained local alignment. Ultimately, the integration of all modules achieved an mR of 40.70%.To further explore the impact of global and local alignment, we conduct additional ablation experiments by varying the weights of global and local alignment, as shown in Table 3 . We maintain the sum of the weights for global and local alignment to be 1. According to the results, the combination of global and local alignment performs best when the weights are set to 0.6 and 0.4, respectively. In this configuration, the mR reach 40.70%, which is 0.60% higher than utilizing only global features and 0.51% higher than utilizing only local features. This demonstrates that global and local information complement each other, allowing for fine-grained contrastive learning to better capture and distinguish the details within remote sensing images. Image-to-Text Retrieval Image-to-Text Retrieval mR R@1 R@5 R@10 R@1 R@5 R@10 0.1 21. 23 4 . Implicit Relation Reasoning (IRR) [12] utilizes a masked language modeling (MLM) approach similar to BERT [8] to implicitly aggregate vision and text features, yielding promising results. However, we argue that this random masking approach may not effectively capture key concepts, as common words like "is" and "a" are also masked. Predicting such words do not significantly contribute to the ability of truly subtle distinctions in text. Thus, we first devise a Masked Attribute Modeling (MAM) module specifically tailored for attribute words, as these terms often offer more discriminative information in retrieval tasks. We annotate and mask all adjectives in the corpus to predict them. In the experiment, we find that this MAM module results in a 0.46% improvement in mR compared with IRR. Subsequently, we further propose our KER module targeted at key concepts. Finally, compared with IRR, our results demonstrate a 1.47% increase in mR, highlighting the importance of key concepts in RSITR tasks and the effectiveness of KER module.Hyperparameter on Objective Function. As shown in Table 5 , we further investigate the hyperparameters of the loss function. While keeping the ratio of global and local loss functions constant, we adjust the weight of the MLM component. The results indicate that the optimal weight for the MLM loss component is 0.5. It is important to note that the weight assigned to MLM should not be excessively high, as the primary focus of the model remains on the process of contrastive learning, and predicting keywords serves as a secondary objective. Image-to-Text Retrieval Image-to-Text Retrieval mR R@1 R@5 R@10 R@1 R@5 R@10 threshold for filtering the global similarity for the next epoch, it effectively filters out noisy sample pairs. However, when the drop ratio is low, the EBA strategy filters out too few samples, failing to effectively eliminate noise. Conversely, when the drop ratio is high, a large number of normal samples are filtered out, leading to suboptimal performance. Therefore, based on these findings, we determine the drop ratio to be 1%, striking a balance between noise elimination and retaining sufficient samples for effective training.3.5.6 Ablation on the Quantity of Keywords for Each Dataset. We further conduct an ablation study on the number of keywords obtained through word frequency analysis on the dataset, as shown in Table 6 . Here, "Word" represents the number of words selected based on the highest frequency for each dataset. "Total" indicates the total number of unique keywords obtained by merging and removing duplicates across all three datasets. The results suggest that the number of words selected for the keyword list should be within an appropriate range. Ultimately, we select the top 512 words with the highest frequency for each dataset as the keyword list.In our comparison of inference time across various approaches, we conduct traditional and CLIP-based methods across the three datasets with a single NVIDIA GeForce RTX 4090 GPU. The "IT" in table represents Inference Time. Note that the GaLR [41] method are not replicated on the NWPU dataset due to a lack of detailed 1.There is a row of red roofed houses near the airport .2.There is a row of red roofed houses near the airport .3.The airport has a lot of white planes . 4.There are a lot of bare land around the airport . 5.A tall airport was built on the land .1.Green clear sea and light yellow beach .2.The beach with khaki sand and the color of the seawater is green .3.The beach with khaki sand and the color of the seawater is green . 4.Green water and sandy beaches . 5.The waves of the green seawater wash the beach .A straight freeway goes through the lawn .There are many buildings on both sides of the railway station.There is a long blue building.A viaduct in the shape of figure of eight is on the bareland . information regarding the additional Ppyolo extractor [22] it employs. As shown in Table 7 , the results indicate that our model lags behind that of the RemoteCLIP, which relies on excluding global features, in terms of inference speed. This discrepancy arises from the integration of fine-grained local alignment in our model, leading to a requirement for increased inference time. Despite this, our proposed EBAKER outperforms RemoteCLIP by 5.30%, 3.84%, and 3.12% in terms of performance, albeit with an increase in inference time by 146%, 78%, and 218%, respectively. Nevertheless, when compared with traditional methods such as KAMCL [11] and GaLR [41] , we exhibit significant advantages in both performance and inference time.Figure 6 displays the top 5 retrieval results, with each row showing the results on the RSICD [23] , RSITMD [38] , and NWPU [5] datasets, respectively. As depicted, our proposed EBAKER successfully retrieves the target images under given queries. Additionally, examining the incorrect retrieved results in the first and third rows reveal that they are actually highly relevant to the query image. This suggests that weakly correlated image-text pairs not only exist in the train set but also in the validation and test sets. Therefore, such errors do not necessarily indicate that the model has failed to learn effective representations. In the future, we will investigate how to address such issue in the test set. 
conclusion: 
 In this paper, we have introduced a novel Eliminate Before Align strategy with Keyword Explicit Reasoning framework (EBAKER), aiming at achieving the transfer of foundation model to remote sensing image-text retrieval model through a one-step fine-grained training. This framework incorporates an Eliminate Before Align strategy to eliminate weakly correlated pairs, thereby promoting the accuracy of fine-grained contrastive learning. Moreover, we employ a Keyword Explicit Reasoning module, which boosts the discriminative ability to the model by predicting nuanced differences in key concepts. Finally, we conduct comprehensive experiments to validate the superiority and effectiveness of our approach on three public benchmark datasets: RSICD, RSITMD, and NWPU.Our method represents a novel attempt to skip the stage of RS pretraining, providing a promising solution for transferring large models directly to other tasks in the remote sensing domain. By tailoring the approach to the characteristics of different downstream tasks, we potentially save a significant amount of additional training data in various domains. Thus, it also lays a foundation for extending our method to other domains, such as product search and pedestrian retrieval for future development. In the future, our efforts will persist in researches on enabling models to adaptively filter data and pay attention to more fine-grained information, aiming at exploring the optimal application of multimodal foundation models in downstream tasks.  
 
 Appendix:  


Decision: This submission to ACM MM underwent meticulous evaluation by three expert reviewers who have all recommended acceptance. The consensus summary of their assessments follows:

Innovativeness and Contributions: The reviewers commend the novelty and efficacy of the proposed method, which makes significant strides in addressing Remote Sensing Image-Text Retrieval. Its robust theoretical underpinnings and empirical results that surpass current methodologies demonstrate a meaningful addition to the field.
Technical Depth and Experimental Validation: The paper presents thorough technical details, rigorous experimental designs utilizing appropriate datasets, and comprehensive comparative analyses. The reviewers appreciate the strength of the experimental outcomes in supporting the authors' claims and applaud the provision of code and data, enhancing reproducibility and future research potiential.
Writing Quality and Clarity: Structurally sound and logically coherent, the manuscript is well-argued with effective use of figures and supplementary materials to augment explanations. While minor suggestions for refinement or additional contextual information were mentioned, these do not detract from the overall readability or comprehension.

Limitations and Recommendations for Improvement:
Despite minor critiques and suggestions for enhancements, such as questions of Reviewer K6aM and zczz, these are viewed as avenues for future work rather than barriers to acceptance. The authors have pledged in their response to consider these recommendations for refinement.

Conclusion and Recommendation:

Given the paper's innovativeness, technical depth, and potential impact on the fields of computer vision and pattern recognition, coupled with the unanimous positive feedback from all three reviewers, this meta-review recommends the acceptance of the paper for presentation at ACM MM, as a poster. We anticipate that the findings will spark keen interest and foster productive discussions among conference attendees, thereby contributing to advancements in the domain.

Reviews: review: paper_summary: paper_summary: This article proposes a new method for the RSITR task. The results on three datasets demonstrate the effectiveness of this method.
strength_weakness: strength_weakness: 1、To address two issues in the RSITR task, the authors proposed two different modules to solve them（i.e., EBA, KER）, and the experimental results demonstrated the effectiveness of these modules.
2、The authors conducted thorough experiments, with detailed ablation studies
limitations: limitations: The innovations in this article are relatively few. The problems to be solved are not clearly articulated, and the proposed solutions are not sufficiently novel.
2、There is a lack of introduction to related work.
3、Regarding Table 1, what is the purpose of using three datasets as the training set simultaneously? Is it just to improve the model's metrics? Since some images in the RSITMD dataset are sourced from RSICD, could there be a data leakage issue?
4、There are writing issues in the article, and some formulas and symbols need to be carefully checked again.
paper_summary: paper_summary: The manuscript introduces the "Eliminate Before Align with Keyword Explicit Reasoning" (EBAKER) framework for Remote Sensing Image-Text Retrieval (RSITR). The proposed approach try to address the challenges encountered when adapting foundation models like CLIP to remote sensing tasks. These challenges primarily include the management of weakly correlated sample pairs and the lack of fine-grained differentiation. EBAKER's "Eliminate Before Align" strategy selectively filters out these weakly correlated pairs before the alignment phase. This aims to enhance the alignment within optimal embedding spaces for more accurate retrieval. Additionally, it incorporates a "Keyword Explicit Reasoning" module designed to highlight key concept differences within remote sensing images,  improving the semantic accuracy of the model. Experiment results demonstrate that the proposed method achieves better performance than previous methods.
strength_weakness: strength_weakness: Regarding the "Eliminate Before Align (EBA)" method, this manuscript introduces the EBA strategy as a preprocessing step aimed at enhancing the quality of the training dataset. By proactively filtering out weakly correlated image-text pairs before the alignment phase, the strategy focuses training on more potent and relevant pairs. This selective filtering is meticulously designed to diminish the influence of noise and improve the alignment accuracy within the training process. Such purification not only optimizes the embedding spaces but also sharpens the model's focus.

After the initial elimination phase, the EBAKER framework employs the “Keyword Explicit Reasoning (KER)” module. This module is adeptly tailored to enhance the semantic understanding of the content by emphasizing key textual concepts that are crucial for accurate image-text pairing.  By focusing on the nuanced aspects of textual content associated with images, KER actively identifies and underscores significant concepts within the texts. This focused approach allows for a more refined and contextually aware alignment process.
limitations: limitations: 1.The method proposed in this manuscript incrementally builds on existing techniques, resulting in insufficient innovation.

2.This manuscript presents the "Eliminate Before Align" (EBA) strategy, which aims to enhance remote sensing Image-Text retrieval by removing weakly correlated pairs from the training dataset. The method uses a global similarity measure to filter Image-Text pairs. However, this strategy may mistakenly exclude useful pairs. Specifically, some Image-Text pairs are weakly correlated overall, but may have locally correlated features that are important for training.

3.This manuscript presents the KER method to enhance the model's focus on critical terms for describing remote sensing images, aiming to improve semantic understanding and retrieval performance. However, the critical limitation is that the effectiveness of keyword selection is depend on the dataset used. If the dataset lacks diversity and does not encompass a broad spectrum of remote sensing contexts, there is a risk that the selected keywords may not perform well when generalized to other datasets or practical remote sensing applications.
paper_summary: paper_summary: In the paper, the problem of strong interclass similarity in remote sensing domain is addressed through the textual perspective, thus an Eliminate Before Align strategy with Keyword Explicit Reasoning framework is proposed to over-screen weakly correlated pairs of samples and reduce their deviation from the optimal embedding space. The effectiveness of the method on multiple datasets is demonstrated experimentally.
strength_weakness: strength_weakness: 1. This work addresses the effect of negative samples in remote sensing multimodal alignment from the perspective of text, by extracting keywords in the text for Mask to eliminate the effect of negative samples before alignment, and the experimental effect is also relatively significant, validated on multiple datasets.

2. An innovative pre-alignment elimination (EBA) strategy is also designed. Being able to automatically eliminate positive sample pairs with low global similarity before comparison, this type of work can theoretically be extended to more remote sensing related application scenarios.
limitations: limitations: 1. Regarding 2.1.3 Keyword Statistics and Mask Generation, there is a lot of similar work in the image-text retrieval domain, can you elaborate on the need for it? Will it be different in the field of remote sensing?

2. There may be important related methods that are not referenced:

[1] Ji, Zhong, et al. "Knowledge-Aided Momentum Contrastive Learning for Remote-Sensing Image Text Retrieval." IEEE Transactions on Geoscience and Remote Sensing 61: 1-13.

[2] Ma, Qing, et al. "Direction-oriented visual-semantic embedding model for remote sensing image-text retrieval." IEEE Transactions on Geoscience and Remote Sensing.

3. In Equation 6, the combination of global and local features is mentioned, where alpha and beta's are determined how? I don't seem to see specific instructions in the article.

4. In Equation 9 and Equation 10, how does the threshold Th get how to set it?