Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-20 14:02:04

Abstract: Scene text retrieval aims to find all images containing the query text from an image gallery. Current efforts tend to adopt an Optical Character Recognition (OCR) pipeline, which requires complicated text detection and/or recognition processes, resulting in inefficient and inflexible retrieval. Different from them, in this work we propose to explore the intrinsic potential of Contrastive Language-Image Pre-training (CLIP) for OCR-free scene text retrieval. Through empirical analysis, we observe that the main challenges of CLIP as a text retriever are: 1) limited text perceptual scale, and 2) entangled visual-semantic concepts. To this end, a novel model termed FDP (Focus, Distinguish, and Prompt) is developed. FDP first focuses on scene text via shifting the attention to text area and probing the hidden text knowledge, and then divides the query text into content word and function word for processing, in which a semantic-aware prompting scheme and a distracted queries assistance module are utilized. Extensive experiments show that FDP significantly enhances the inference speed while achieving better or competitive retrieval accuracy. Notably, on the IIIT-STR benchmark, FDP surpasses the state-of-the-art method by 4.37% with a 4 times faster speed. Furthermore, additional experiments under phrase-level and attribute-aware scene text retrieval settings validate FDP's particular advantages in handling diverse forms of query text.

Main paper: 
 related work 2.1 scene text retrieval: 
 Most of the early STR approaches tend to follow the OCR pipeline [1, 26, 33] . They first take two separate steps of text detection and recognition to extract words in each image, and then match these words with the query word for retrieval. For instance, Mishra et al. [21] first investigate the STR task, proposing to rank all images based on the ordering and positioning of localized characters. Jaderberg et al. [11] perform text spotting with a CNN network and evaluate the occurrences of the query word within the spotted words. However, those straightforward attempts could not obtain satisfactory performance and are also not efficient. To solve this problem, Gomez et al. [8] leverage a compact representation named Pyramidal Histogram of Character (PHOC) [2] and propose a single-shot CNN architecture that simultaneously predicts text proposals and corresponding PHOCs. In this way, the STR task can be completed by a simple nearest neighbor search. Considering current handcraft representations (including PHOC) still cannot well reflect the distance between text and image modalities, recent methods are dedicated to mining better similarity measures. TDSL [27] establishes an end-to-end network that jointly optimizes text detection and cross-modal similarity learning. To mitigate the gap across different modalities, Wen et al. [30] propose to cast STR as an image-to-image matching problem. Although better retrieval accuracy is achieved, it comes at the cost of inference speed.Vision-language models pre-trained on web-scale data have been demonstrated to exhibit certain OCR capabilities [10, 17, 18, 32, 35] . As reported in [23] , the CLIP model shows favorable OCR performance in rendered text images. To further mine the underlying rationales, [14] thoroughly inspects different versions of CLIP. This work uncovers that CLIP suffers from severe text spotting bias because many captions in CLIP's training dataset tend to parrot the visual text embedded within images. Through orthogonal projections of the learned representation space into "learn to spell" and "forget to spell" parts, [20] disentangles such bias to some extent. Besides, LoGoPrompt [24] finds that synthetic text images are good visual prompts for vision-language models, which can be used to improve image classification performance.Inspired by these observations, several works aim to enhance OCR tasks by transferring the knowledge from CLIP. In the field of scene text recognition, CLIP4STR [36] designs a two-branch framework in which the recognition results are predicted by the visual branch and then refined by the cross-modal branch. CLIP-OCR [29] resorts to the knowledge distillation technique and guides the recognition with both visual and linguistic knowledge from CLIP. In the field of scene text detection, TCM [34] integrates CLIP with existing text detectors, leading to obvious performance improvements in domain adaptation and few-shot capabilities. However, CLIP merely acts as an auxiliary module in these works. Whether it is possible to turn CLIP directly into a scene text reader (spotter or retriever) remains an unexplored problem. 
fdp method: 
 The overview of the proposed FDP framework is illustrated in Fig. 3 . Given a query text (𝑄 = "house"), FDP fulfills the STR task with a pipeline of "Focus, Distinguish, and Prompt".Considering CLIP is pre-trained on conventional image-text pairs and thus lacks fine-grained awareness of visual text information, the first step of FDP is directing CLIP to focus on scene text. To be specific, for each image from the gallery, we first square it to obtain the input image 𝐼 𝑖𝑛𝑝𝑢𝑡 ∈ R 𝐿×𝐿 (𝐿 is the image length), i.e., perform zero-padding to make the shorter side match the longer side. The goal is to avoid the loss of image content caused by the center cropping operation during CLIP's preprocessing. Then, the frozen ResNet-based vision encoder of CLIP is employed to extract the global feature 𝑓 𝑔𝑙𝑜𝑏𝑎𝑙 ∈ R 𝐶 ×𝐻 ×𝑊 of 𝐼 𝑖𝑛𝑝𝑢𝑡 , where 𝐶, 𝐻 and𝑊 stand for the channel, height and width dimensions respectively. Based on this global feature, two modules including dynamic attention shift and text knowledge probing are proposed to highlight scene text information and address the limited text perceptual scale problem. Dynamic Attention Shift. The limitation of input resolution is an intractable problem encountered by pre-trained vision-language models. It greatly impairs scene text understanding performance because text often occupies a very small part of the image. Existing efforts resolve this problem by subdividing into image patches [12] , retraining a vision encoder [3] , or processing in the frequency domain [5] , which are not efficient. Instead, in this work we find that it may be enough to give the model a glimpse of the rough area where text is clustered. To this end, we employ text detection supervision to train a lightweight text localization network, and then utilize the normalized probability map to reweigh the features in the average pooling layer. Specifically, as the multi-head attention layer in CLIP's vision encoder loses the 2D image information, we first introduce a reformulated head following [37] to recover the 2D convolutional image feature 𝑓 𝑐𝑜𝑛𝑣 ∈ R 𝐸×𝐻 ×𝑊 , where 𝐸 is the embedding dimension in CLIP. Then, the localization probability map 𝐼 𝑙𝑜𝑐 ∈ R 𝐻 ×𝑊 is obtained via a learnable convolutional layer. We train the text localization network via a class-balanced crossentropy loss, given by:EQUATION)where 𝑌 is the ground-truth localization map generated by the text detection annotation, and 𝛽 is a balancing factor which is defined as:EQUATIONAfter that, the predicted localization probability map is adopted as a new attention mask to dynamically refine the attention applied to the global feature. The details of the dynamic attention shift module are illustrated in Fig. 4 . CLIP uses Transformer-style multi-head attention to perform average pooling, where the 2D global feature is first flattened into a 1D sequence and then generates a key-value pair to interact with the globally average-pooled feature (query). Consequently, the localization probability map is also flattened into a 1D sequence and then weights the global feature at each spatial location. The derived attention feature 𝑓 𝑎𝑡𝑡𝑛 ∈ R 𝐸 can shift the model attention to the scene text area. Text Knowledge Probing. Empirically, we find that when we query CLIP with the word "house", it is possible to return the corresponding object (an image of a house) instead of the scene text (an image says "house"). This is because the neurons in CLIP's vision encoder tend to activate on the whole image rather than specific text information. Therefore, we consider whether we could design a simple strategy to probe the text-related knowledge hidden in CLIP. Drawing inspiration from previous work [23] that conducts zero-shot image classification using a predefined template "a photo of [CLS]", we propose to utilize the plain text "scene text" as a probe and obtain its language embedding as the probe feature 𝑓 𝑝𝑟𝑜𝑏𝑒 ∈ R 𝐸 , 69.79 78.17 which is then interacted with 𝑓 𝑎𝑡𝑡𝑛 . Since the representations of vision and language are well-aligned in the embedding space of CLIP, this probe will naturally turn CLIP into a model that is more sensitive to scene text. Subsequently, the interacted feature is fused with the attention feature 𝑓 𝑎𝑡𝑡𝑛 as the image feature 𝑓 𝑖𝑚𝑔 to comprehensively encode the image for retrieval. The text knowledge probing process is formulated as:𝑓 𝑖𝑚𝑔 = MHCA(Q = 𝑓 𝑎𝑡𝑡𝑛 , K = 𝑓 𝑝𝑟𝑜𝑏𝑒 , V = 𝑓 𝑝𝑟𝑜𝑏𝑒 ) + 𝑓 𝑎𝑡𝑡𝑛 (3)where MHCA means the multi-head cross-attention mechanism.Several works [14, 24] have revealed that the CLIP model exhibits inherent bias towards visual text, e.g., an image of "dog" may be recognized as "cat" by placing the text that says "cat". The reason is that the captions CLIP pre-trained with are often simple repetitions of text embedded in images. In this work, we further observe that this bias is essentially the entanglement between visual and semantic concepts. To be specific, we select 500 words with the highest frequency from the MLT [22] training set, and then group their CLIP language embeddings into two clusters via K-Means. The t-SNE visualization result is depicted in Fig. 5 (a). As can be seen, these words naturally fall into two clusters, namely, the content words and the function words. Between them, the content words have explicit semantics, usually appearing next to the thing they represent, so they exhibit strong visual-semantic entanglement. In contrast, the function words generally appear in the captions as conjunctions, which do not correspond to specific concepts. To investigate this effect on STR, we evaluate the retrieval accuracy of content words and function words respectively by applying the frozen CLIP models on SVT [28] dataset, as shown in Fig. 5 (b). It is obvious that for the three CLIP models with different capacities, the mAP scores on function words are significantly lower than those on content words. This inspires us that different retrieval solutions should be taken on these two clusters. Thus, before each retrieval process, we pre-distinguish whether the given query is a content word or a function word. Without the need for specialized tools, this can be easily achieved via unsupervised clustering, namely predicting which K-Means cluster the query text belongs to.Prompt tuning is a promising paradigm that aims to adapt the knowledge from a pre-trained model to a target domain [15] . Mirroring the success of prompt tuning in natural language processing and cross-modal learning, we leverage it to facilitate CLIP for efficient text retrieval.Semantic-aware Prompting. To improve retrieval performance on both content words and function words, we develop a semanticaware prompting scheme. Inspired by CoOp [38] , we introduce two sets of learnable context vectors to serve the retrieval of content words and function words respectively. Formally, the text prompts fed to the frozen CLIP language encoder are organized as:EQUATIONEQUATIONwhere 𝑃 𝑐 and 𝑃 𝑓 denote the prompts for content words and function words respectively. 𝑀 and 𝑁 are the length of learnable context vectors, and 𝑄 represents the query text. Then, the CLIP language encoder outputs the prompt feature 𝑓 𝑝𝑟𝑜𝑚𝑝𝑡 . We calculate the cosine similarity between 𝑓 𝑖𝑚𝑔 and 𝑓 𝑝𝑟𝑜𝑚𝑝𝑡 to measure the pairwise similarity score between the input image and query text, i.e., 𝑆 (𝐼, 𝑄). The symmetric cross-entropy loss L 𝑎𝑙𝑖𝑔𝑛 over a batch is applied to contrastively align the matched (𝐼, 𝑄) pairs. Distracted Queries Assistance. Due to the limited input resolution and the visual-semantic entanglement, CLIP can perceive text to some extent, but it indeed lacks the ability of fine-grained character discrimination. To address this problem, a distracted queries assistance module is proposed, which teaches the FDP model to better recognize text during training. In particular, for a query text 𝑄, we utilize a dictionary to generate 𝐾 (set to 5) distracted queries 𝑄 1 , 𝑄 2 , ...𝑄 𝐾 that have the smallest edit distances with 𝑄. These distracted queries are taken as hard negative samples which are also converted to text prompts and fed into the CLIP language encoder, predicting similarities scores 𝑆 (𝐼, 𝑄 𝑘 ), 𝑘 = 1, ..., 𝐾. Meanwhile, the edit distance of each distracted query from the ground-truth query 𝐷 (𝑄, 𝑄 𝑘 ) is calculated. Subsequently, we convert these 𝐾 sets of similarity scores and edit distances into probability distributions and compute the KL divergence between them as the training loss L 𝑑𝑖𝑠𝑡𝑟𝑎𝑐𝑡 . The objective is to maximize the similarity scores of the distracted queries that are close to the ground-truth while minimizing the similarity scores of the distracted queries that are far from the ground-truth. The proposed FDP is trained with the following loss functions:EQUATION)where 𝜆 1 , 𝜆 2 , and 𝜆 3 are used to balance these losses, which are all set to 1 in our implementation.During inference, the distracted queries assistance module is removed. Given a query text 𝑄, images in the gallery are ranked according to the predicted similarity score 𝑆 (𝐼, 𝑄). When extending our FDP model to phrase-level or attribute-aware scene text retrieval settings, 𝑄 is directly assigned the corresponding form of query, and the inference process remains unchanged. 
experiments 4.1 datasets: 
 IIIT Scene Text Retrieval (IIIT-STR) [21] is a popular benchmark that contains 10000 images and 50 predefined queries. The images are collected using Google image search, so this dataset has a large variability in text fonts, styles, and viewpoints. Street View Text (SVT) dataset [28] is a collection of natural street scenes. It consists of 100 training images and 249 testing images. All annotated words (427 words) in the test set are employed as the query text. TotalText [4] is a scene text dataset consisting of 1255 training and 300 testing images. The 60 words that occur most frequently in the test set are selected as queries. Multi-lingual Scene Text (MLT)-Eng is the English subset of MLT [22] , which includes about 5000 images of natural scenes.In our experiments, MLT-Eng is only used for training the proposed model. IIIT-STR, SVT, and TotalText are the testing datasets. It should be noted that as CLIP's potential is fully explored, 900k synthetic training images used in [27, 30] can be saved in FDP.The query terms in existing datasets are all single words. To validate whether the STR models can be generalized to arbitrarylength query text, we introduce a new Phrase-level Scene Text Retrieval (PSTR) dataset. To build it, we select 36 phrases that occur frequently in life as queries, each containing 2 to 4 words, e.g., "school bus", "handle with care". For each query, we collect 15 images from TextOCR dataset [25] and Google image search respectively. In total, PSTR includes 1080 images and 36 query text.Based on CLIP with different capacities, we build several versions of FDP models, as summarized in Tab.1. As the input image size supported by CLIP is very limited, we expand the image size in FDP. However, directly expanding the image size makes the position embedding inherited from CLIP incompatible. To tackle it, we propose a new learnable position embedding whose parameters are initialized with the nearest interpolation of original parameters. We optimize FDP using Adam optimizer with an initial learning rate of 2e-3. The batch size is 48, and the number of training epochs is about 8. For fair comparisons, our experiments are implemented with Pytorch. All FDP models are trained on an NVIDIA A6000 GPU and tested on an NVIDIA 1080 GPU.In this section, we compare FDP with existing methods on three STR benchmarks, i.e., IIIT-STR, SVT, and TotalText. As a task to pursue practical applications, the inference speed of STR is undoubtedly very important, while previous methods are subject to the balance of retrieval accuracy and inference speed. In this paper, we first investigate employing the frozen CLIP model directly as the retrieval engine. As reported in Tab.2, it is surprising that CLIP already exhibits some retrieval capabilities even though it has not been specially trained on STR tasks. In particular, CLIP-RN50 obtains 52.93% and 65.07% mAP scores on the IIIT-STR and SVT datasets respectively, which even exceeds several dedicated STR models [9, 21] at a much faster speed (76.32 FPS).Based on this observation, FDP is proposed to better unleash CLIP's potential for the STR task. On the IIIT-STR benchmark, we can notice that FDP-S initialized with the CLIP-RN50 base model boosts the mAP score by 28.84% (52.93%->81.77%), achieving an appealing result of 81.77%. Meanwhile, the inference speed is also superior (45.11 FPS), even faster than PHOC-based methods [8, 19] . When upgrading the model to a large size, FDP-L significantly outperforms the competitive model [30] by 12.06% (77.40%->89.46%) at a comparable speed. Compared with IIIT-STR, the query terms of SVT and TotalText contain many function words and often occupy small areas in images, which are more challenging for STR models. Nevertheless, even without complicated network design, FDP also reaches competitive performance on these datasets. To further boost the retrieval accuracy, we attempt to integrate the subdivision enhancement strategy here, i.e., subdividing the input image into 4 equal patches and combing the outputs of these patches. The mAP scores are improved by 2.03%, 1.55% and 2.84% on IIIT-STR, SVT and TotalText, outperforming existing STR methods.To provide intuitive analyses of FDP in comparison with previous methods, a typical example is visualized in Fig. 6 (a). Given the query word "adobe", TDSL relies entirely on character composition From the rank@7 and rank@10 images retrieved by FDP-S, we notice that the proposed method can recall images where the query text is not so salient.Overall results. In Tab.3, a detailed ablation experiment is conducted to verify the effectiveness of each module. We start by training a model that only utilizes the new learnable position embedding, whose mAP scores on IIIT-STR and SVT are 75.74% and 79.97% respectively. It reveals that enlarging the image size (i.e., enhancing the text perceptual scale) is of critical importance for STR. Based on this, we gradually add the proposed modules and observe that each module brings noticeable improvements. In the "Focus" step, the dynamic attention shift and text knowledge probing modules can be considered to highlight scene text information from visual space and semantic space respectively. They bring 2.64% and 0.55% gains on the IIIT-STR dataset, which are proved to be effective. In particular, as IIIT-STR contains a large number of images without any text, the "Focus" step has a more significant effect on the IIIT-STR dataset than on the SVT dataset. Then, we study the effect of different prompt strategies. When simply adopting the learnable prompt method in [38] (#4), the mAP scores reach 80.07% and 81.03% on these two datasets. In contrast, we claim that content words and function words should be distinguished and exploit different customized prompts. Following this idea, our semantic-aware prompting scheme improves the performance to 81.27% and 81.94%. Further, by adding the training strategy of distracted queries assistance, 81.77% and 82.56% mAP scores are finally obtained. Analysis of the predefined probe. The goal of the predefined probe is to stimulate the text-related knowledge hidden in CLIP. In Tab.4, we conduct an ablation study of the predefined probe on the IIIT-STR benchmark. The results show that if the predefined probe is removed, the mAP score decreases from 81.77% to 79.58%. Furthermore, different strings are utilized to generate language embeddings that interact with the image attention feature. Compared to the "without predefined probe" baseline, these text-related probes can enhance the performance. Among them, "scene text" contributes to the best accuracy, implying that in CLIP's training data, the plain text "scene text" may appear frequently with the scene text content from images. Analysis of the context length. In the semantic-aware prompting module, the hyperparameters 𝑀 and 𝑁 determine the context length for content words and function words respectively. In Tab.5, we evaluate the model performance on the SVT dataset to analyze the effect of these hyperparameters. According to the results, FDP reaches the best performance when 𝑀 = 4 and 𝑁 = 2. It may suggest that function words contain less semantics than content words, so they do not require complicated descriptions of context. Analysis of the number of distracted queries. In the distracted queries assistance module, 𝐾 distracted queries are generated to help the model identify similar words. The ablation results of 𝐾 are reported in Tab.6. From them, we can see that a smaller 𝐾 may weaken the discrimination ability of FDP, while a larger 𝐾 will introduce many negative samples that are far from the query. Thus, 𝐾 is set to 5 in our experiments.Phrase-level scene text retrieval. In reality, the scene text that people expect to retrieve is often of arbitrary length, such as "ice cream", "do it yourself ". To validate the generality of our method over arbitrary-length query text, we evaluate FDP and several STR models on PSTR. For the comparison models [8, 27] , since they can only accept word instances, we split each phrase into words, search them separately, and then average the corresponding similarity scores. As shown in Tab.7, FDP is more flexible than existing STR models in phrase-level retrieval. Specifically, the PHOC-based method [8] only achieves 68.01% mAP score on PSTR. We speculate this is because many split words are too short (e.g., "do" in "do it yourself ") to be accurately retrieved by the PHOC-based search algorithm. Although TDSL [27] can get 89.40% with the simple splitting operation, it is inherently flawed due to the local retrieval mechanism. From Fig. 6 (b), we can see that for the query text "no smoking", TDSL may return images containing "no engine" (rank@3) or "no softener" (rank@5), which do not meet retrieval expectations. In addition, due to the dense text distribution in the PSTR dataset, these OCR-based comparison models run slower than on IIIT-STR. In"no smoking" contrast, the FDP-S model reaches 92.28% mAP score on PSTR, outperforming existing methods by great margins. More importantly, as FDP does not rely on text detection or recognition processes, the retrieval speed will not be affected. Attribute-aware scene text retrieval. Considering that people often query scene text with fine-grained attributes for more accurate search results, we further explore extending FDP to the attributeaware scene text retrieval setting. We design some attribute-related queries and search corresponding images from the IIIT-STR dataset. Several typical retrieval examples are illustrated in Fig. 7 . These results manifest that the CLIP-based FDP model is naturally suitable for attribute-aware scene text retrieval because it takes advantage of CLIP's prior knowledge. FDP can well deal with attribute-related information such as color, font, and even position of scene text, returning images that users want. Admittedly, this is not available for conventional OCR-based STR models. 
conclusion: 
 In this paper, we explore CLIP's intrinsic potential for efficient and flexible scene text retrieval. An OCR-free retrieval model named FDP (Focus, Distinguish, and Prompt) is proposed, in which the "Focus" design highlights scene text information hidden in CLIP while "Distinguish" and "Prompt" designs further overcome the negative effects caused by visual-semantic entanglement. Experimental results on three datasets demonstrate the effectiveness of our proposed modules and show that FDP achieves a better tradeoff between retrieval accuracy and inference speed. In addition, 
: 
 Figure 1 : Illustration of the trade-off between retrieval accuracy (mAP scores) and inference speed (FPS) on the IIIT-STR benchmark. Our proposed FDP method achieves better balance than previous methods.for decades. Different from common scene text understanding tasks such as text detection, text recognition, and end-to-end text spotting, Scene Text Retrieval (STR) is an emerging topic that only focuses on text of interest, i.e., searching images containing a given query text from an image gallery. As such, STR is beneficial for many applications like product image search, program recommendation, and electronic book archives management [6, 7, 31] .With the aid of Optical Character Recognition (OCR) techniques, STR has made remarkable progress in recent years [9, 13, 16] . Nevertheless, existing methods still suffer from two critical limitations. First, as illustrated in Fig. 1 , there is a dilemma of how to balance retrieval accuracy (mAP scores) and inference speed (FPS). Specifically, most STR models follow the two-stage pipeline that first detects text regions and then compares these regions with the query text for retrieval. In this pipeline, either an exact text detection or recognition process is required, which significantly slows down the inference speed. Comparatively, Gomez et al. [8] achieve fast text retrieval using a single-shot CNN architecture, but it is limited by relatively low retrieval accuracy. Second, in real life, the query text that people expect to retrieve is often in various forms. However, current efforts rely on the local retrieval mechanism that treats word instances as query units, leading to inherent inflexibility in phrase-level or attribute-aware scene text retrieval (see Fig. 2 ).Recently, Contrastive Language-Image Pre-training (CLIP) [23] has become a powerful foundation model for learning cross-modal representations and enabling zero-shot transfer to downstream tasks. More remarkably, several works [14, 24] have demonstrated CLIP also implies OCR capabilities via pre-training on massive image-text pairs. It gives us a new insight: can we explore the intrinsic potential of CLIP for efficient and flexible STR? To this end, we investigate the advantages and deficiencies of CLIP in the STR task through an empirical study. A surprising finding is that simply applying the frozen CLIP can even achieve better accuracy than some dedicated STR models. Moreover, thanks to CLIP's simple network design, the retrieval speed is also superior. Despite these impressive results, there are still two challenges that hinder CLIP from being an ideal retrieval engine: 1) Limited text perceptual scale. As the image resolution input into CLIP is very limited (e.g., 224×224), and scene text usually occupies only a small part of the scene image, a lot of text may be ignored or misrecognized by CLIP.2) Entangled visual-semantic concepts. Due to the prevalence of text in natural images, there is confusion between visual text and semantic concepts in CLIP's cognition [20] . Its specific impact on STR is that the CLIP-based retrieval model performs much better on content words (e.g., "coffee", "hotel") than on function words (e.g., "and", "with") because only content words represent exact semantics. Besides, the model may have difficulty distinguishing similar words (e.g., "advice" and "advise") because their semantics are close in the embedding space.In this paper, we propose a model named FDP (Focus, Distinguish, and Prompt) to tackle the above challenges. Concretely, for each image in the gallery, we firstly force CLIP to focus on scene text by 1) applying the rough text localization results to refine the model attention on images, and 2) leveraging CLIP's well-aligned visionlanguage representations to prob text knowledge. Then, given a query text, we distinguish whether it is a content word or a function word via unsupervised clustering and determine the retrieval solution accordingly. Finally, a semantic-aware prompting scheme is developed, which converts the query text into a learnable prompt and ranks images by computing their similarity scores with each image. In addition, a distracted query assistance strategy is involved during training to resist the negative effects of similar words. Extensive experiments on three benchmarks show that FDP can achieve better or competitive accuracy compared to existing models with a faster inference speed. To further evaluate the effectiveness of STR methods over arbitrary-length query text, we introduce a new benchmark of phrase-level scene text retrieval (PSTR). Meanwhile, qualitative experiments regarding attribute-aware scene text retrieval are conducted. These experimental results demonstrate the generalization and flexibility of FDP.Overall, the main contributions of this work are three-fold: 1) To the best of our knowledge, it is the first work to directly extend CLIP for scene text retrieval. We summarize both the advantages and deficiencies of CLIP in dealing with the STR task and propose a novel FDP (Focus, Distinguish, and Prompt) method.2) In contrast to previous works, FDP steers the prior knowledge from CLIP and eliminates the complicated text detection/recognition process, thus achieving a better trade-off between retrieval accuracy and inference speed. Notably, FDP outperforms the state-of-the-art method [30] by 4.37% mAP score with a 4 times faster speed on the IIIT-STR benchmark.3) We evaluate existing STR methods in phrase-level and attributeaware scene text retrieval settings, further verifying the superiority of FDP in handling diverse forms of query text. 
 
 Appendix: Retrieval Results"galaxy in red background" "coffee in italic font""school written on the wall" FDP can easily generalize to other settings like phrase-level and attribute-aware scene text retrieval, which are more practical for requirements in real scenarios. 


Decision: This paper aims to address the Scene Text Retrieval (STR) problem. The goal of STR is to return all images that contain the query text. The authors argue that most of the existing STR approaches conduct OCR to extract text from images and then conduct the similarity search, which could limit the efficiency. Towards this end, the authors propose an FDP model, which firstly leverages CLIP to roughly localize the text in the image, using k-means to split the query into ‘content’/ ‘function’, and explore the learnable semantic embeddings to enhance the query.

Most of the reviewers consider the application of CLIP in STR task interesting. The proposed FDP module can mitigate some of the drawbacks of CLIP in STR tasks, which enhances the technical contribution of this work. 
There are a few issues that were pointed out by reviewers, including further clarifying experimental settings, discussion on failure cases of FDP, and typos. I would also request authors to report full results of “CLIP with large positional embedding”. As can be seen from table 3, it greatly improves the vanilla clip results and serves as a strong baseline. Importantly, the reviewer has commented that the paper should consider adding further related works on scene text recognition/spotting to strengthen the paper.

Balancing the strengths and weaknesses of this work, the overall sentiment among reviewers is positive. Given the innovative application of CLIP and the potential of the FDP model, I recommend a borderline acceptance, contingent upon the authors addressing the aforementioned areas for improvement.

Reviews: review: paper_summary: paper_summary: The paper introduces a new CLIP-based method for scene text retrieval, aiming to find images containing specified query text without relying on OCR. The proposed pipeline uses pre-trained CLIP vision and text encoders, combined with a set of scene text retrieval scpecific trainable modules. First, a rough localization model on top of CLIP visual features and a multi-head cross-attention layer with the CLIP text ambedding for the phrase "scene text"  helps focusing attention on text areas. Second, they use K-Means over CLIP word embeddings to distinguish between different types of queries that leverage different learnable prompts as a kind of learnable query expansion mechanism. Finally, a distracted queries assistance module generates hard negative samples with minimal edit distances to the query and uses the KL divergence between CLIP similarity scores and edit distances as a training loss.
strength_weakness: strength_weakness: + The proposed method achieves new state of the art results on several benchmarks.

+ The use of CLIP for scene text retrieval tasks is an unexplored approach that makes a lot of sense.

+ A new Phrase-level Scene Text Retrieval (PSTR) dataset is proposed, in which queries have more than on word.
limitations: limitations: - The paper does not clarify the source of the ground truth for training the rough localization model within the Dynamic Attention Shift module.

- The training data used for the KMeans module is not specified, and the decision to use only two clusters (k=2) is not well-justified. It is possible that using more clusters could yield better results, and this aspect should be further explored and discussed.

- The use of the Distracted Queries Assistance module implies modifications to the standard CLIP loss function, but the details provided are insufficient in my opinion. Specifically, it is unclear how the distractors for one query interact with the images relevant to other queries within a batch, and a more detailed explanation of the loss computation at the batch level is needed.

- The implications of distractors with occluded words in a given image, which might be relevant to the retrieval task, are not addressed. This oversight could impact the model's performance in practical scenarios where text occlusion is common.

- There are unclear statements in the Implementation Details section:
    "We expand the image size in FDP. We propose a new learnable position embedding whose parameters are initialized with the nearest interpolation of original parameters."
    "It should be noted that as CLIP’s potential is fully explored, 900k synthetic training images used in [27, 30] can be saved in FDP."

- The paper uses the RN50 version of CLIP rather than the ViT versions, which generally provide better results in other tasks. The rationale behind this choice is not explained, and comparing the performance of different CLIP versions could provide insights into the model's effectiveness.
paper_summary: paper_summary: This paper explores the intrinsic potential of Contrastive Language-Image Pre-training (CLIP) for OCR-free scene text retrieval. Extensive experiments show that FDP significantly enhances the inference speed while achieving better or competitive retrieval accuracy.
strength_weakness: strength_weakness: The authors propose a novel model termed FDP (Focus, Distinguish, and Prompt) is developed. FDP first focuses on scene text via shifting the attention to the text area and probing the hidden text knowledge, and then divides the query text into content word and function word for processing, in which a semantic-aware prompting scheme and a distracted queries assistance module are utilized. 
This paper is easy to understand. The description of the core idea is clear.
limitations: limitations: The notation in Table 2 is wrong.
The case study is not good enough.
It is better to pay more attention to show the superiority of OCR-free methods. The accuracy is not a good aspect.
paper_summary: paper_summary: The authors present FDP, short for Focus, Distinguish, and Prompt. FDP explores the potential of CLIP for scene-text retrieval, as previous work has proven that this model can recognize text (to some extent). However, CLIP is a vision and language foundation model and it is not explicitly trained to read. This means that, even though the model can perform scene-text retrieval, the entanglement between object and text features makes the retrieval results suboptimal. The three main parts of the pipeline (Focus, Distinguish, and Prompt) try to disentangle the object/background semantics and textual features and improve scene-text retrieval.

The "Focus" part of the network takes the image features from CLIP's vision encoder and highlights the text information of the image features, which results in a feature vector with richer textual features. The "Distinguish" part tries to help the model differentiate between content and function words, which further helps the disentanglement between visual and semantic concepts. The "Prompt" part processes the query (the text to be retrieved) and tries to align the ground truth image representation $I$ with the query representation $Q$. The authors also use a series of distractor queries $Q^k$ which they try to distance from the image representation $I$ and the query representation $Q$.

Furthermore, the authors introduce the PSTR dataset, a scene-text retrieval test set that features phrase-level queries, instead of words. The nature of the CLIP text encoder allows the authors to prompt the model with these phrases (or even whole sentences, but I imagine that is not as useful), as opposed to other retrieval models which can only process words.
strength_weakness: strength_weakness: The model appears to successfully adapt CLIP for scene-text retrieval. The different components of the network seem to have been carefully crafted to mitigate the biases that CLIP learned as a vision and language model. Even though plain CLIP already obtains decent scores (row 1 from Table 3), each component seems to help FDP focus on the textual features learned by CLIP.

The three versions of FDP (Small, Base, and Large) obtain better results in the IIIT-STR dataset than the SOTA, and the Small version manages to surpass all models in inference speed. However, the Small and Base models seem weaker in the SVT and TotalText datasets. The Large model that does not use subdivision seems to be competitive with [30], and the subdivision model obtains the SOTA in the three datasets, albeit at lower inference speeds.

I am especially interested in the results using phrases, instead of single words. Using CLIP's text encoder seems like a good way of expanding the scope and applications of scene-text retrieval. There is also uncovered potential in the "attribute-aware" retrieval, of which the authors only show qualitative results.
limitations: limitations: I am slightly worried about how much the image resolution helps the retrieval performance of FDP (a jump of 13\% on IIIT-STR and 14\% on SVT between CLIP-RN50 and base FDP-S from row 1, Table 3), a product of the limitations of CLIP when it comes to reading text. It makes me wonder if the failure cases of SVT and TotalText are mainly because of that.

I have some other minor concerns:

- The authors have used the ResNet-based image encoder from CLIP. Have the authors considered the ViT version?

- I would also appreciate a couple of qualitative examples showing the capacity of disentanglement of FDP compared to plain CLIP (model with the learned embedding vectors, row 1 from Table 3), although I understand that the space is limited.

- The Adam optimizer, which is used by FDP, is not cited in line 619.

- Finally, this is not as much of a concern, but I think the "attribute-aware"-based retrieval has the potential to be further explored in future works.
paper_summary: paper_summary: This paper aims to  retrieve the images using scene text based on CLIP model.
strength_weakness: strength_weakness: The strength is to explore CLIP for scene text image retrieval.
limitations: limitations: The work looks good but the following are the major weaknesses of the proposed work. (i) First of all exploring CLIP for scene text recognition and image retrieval is not new as the authors claimed in the paper. There are works that use CLIP for scene text prediction without recognition. The model uses image and textual features for prediction and text spotting in the scene images. There is no discussion on scene recognition and spotting in the paper as they are relevant to the proposed work. Once you recognize and spot the text in scene images, it is just adding a prompt step to retrieve. When we compare the proposed work with the state-of-the-art of scene text recognition and spotting, the novelty is limited. This is because the proposed work is just a combination of the existing models. The results are good but there should be comparative study with the latest existing methods for image retrieval. This is missing in the present form. At the same time, the authors must check the performance of the proposed method with latest CLIP based scene text recognition and spotting. In addition, there are methods for visual question answering. These methods work well for  any query. Again, there is no discussion about VQA methods. The authors must compare the proposed method with the existing VQA methods. Instead of using existing models for scene text detection and recognition, they should have developed their own method. Therefore, the contribution and novelty are limited.

Most important point is that the title says, it is a retrieval problem but the content show, it is a text detection and recognition problem. There are some results in the experimental section but there is no proper step presented in the proposed methodology section. After text recognition or spotting, how the method retrieves the images from the dataset is not clear. What is the difference between text detection, recognition, spotting and retrieval should  be clear in the work. There are many ways to retrieve the images. One way is to use detected text, another way is to use recognition results and another way is spotting. I did not see the proper method and the results to understand how the retrieval problem has been solved. When we split the whole phrase into words and then combine the score, how the method can assure the correct retrieval results. When we look at the retrieval results, there are images without text information. How an image without text can be retrieved. What is the advantage of text detection and recognition? Overall, it looks like the retrieval task is incomplete and there are not sufficient content in the proposed methodology section and the results in the experimental sections to justify the title.