MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recycling
Authors: Liang Yin
Xudong Xie
Zhang Li
Xiang Bai
Yuliang Liu
Venue: NeurIPS 2025 poster ()
Keywords: scene text retrieval
weak-supervised learning
fine-grained image text matching
Primary area: applications
TL;DR: We present a novel box-free method termed MSTAR for multi-query scene text retrieval.

Abstract: Scene text retrieval has made significant progress with the assistance of accurate text localization.
However, existing approaches typically require costly bounding box annotations for training. 
Besides, they mostly adopt a customized retrieval strategy but struggle to unify various types of queries to meet diverse retrieval needs.
To address these issues, we introduce Multi-query Scene Text retrieval with Attention Recycling (MSTAR), a box-free approach for scene text retrieval. 
It incorporates progressive vision embedding to dynamically capture the multi-grained representation of texts and harmonizes free-style text queries with style-aware instructions. Additionally, a multi-instance matching module is integrated to enhance vision-language alignment.
Furthermore, we build the Multi-Query Text Retrieval (MQTR) dataset, the first benchmark designed to evaluate the multi-query scene text retrieval capability of models, comprising four query types and $16k$ images. 
Extensive experiments demonstrate the superiority of our method across seven public datasets and the MQTR dataset. Notably, MSTAR marginally surpasses the previous state-of-the-art model by 6.4\% in MAP on Total-Text while eliminating box annotation costs. Moreover, on the MQTR benchmark, MSTAR significantly outperforms the previous models by an average of 8.5\%. 
The code and datasets are available at \href{https://github.com/yingift/MSTAR}{https://github.com/yingift/MSTAR}.

MSTAR: Box-Free Multi-Query Scene Text Retrieval with Attention Recycling
Liang Yin Xudong Xie
Zhang Li
Xiang Bai Yuliang Liu"
Huazhong University of Science and Technology
(liangyin, xdxie, zhangli, xbai, ylliu)Ohust.. edu. cn
Abstract
Scene text retricval has made significant progress with the assistance of accurate text localization. However, existing approaches typically require costly bounding box annotations for training. Besides, they mostly adopt customized retrieval strategy but struggle unify various types of queries to meet diverse retrieval needs. To address these issues, we introduce Multi-query Scene Text retrieval with Attention Recycling (MSTAR), box-free approach for scene text retrieval. It incorporates progressive vision embedding to dynamically capture the multi- grained representation of texts and harmonizes free-style text queries with style- aware instructions. Additionally, a multi-instance matching module is integrated enhance vision-language alignment. Furthermore, we build the Multi-Query Text Retrieval (MQTR) dataset, the first benchmark designed to evaluate the multi- query scene text retrieval capability of models, comprising four query types and 16k images. Extensive experiments demonstrate the superiority of our method across seven public datasets and the MQTR dataset. Notably, MSTAR marginally surpasses the previous state-of- vfte-of--hhe-at the-art model by 6.4% in MAP on Total-Text while eliminating box annotation costs. Moreover, on the MQTR benchmark, MSTAR significantly outperforms the previous models by an average of 8.5%. The code and datasets are available htttss//gghhu..com/yyggttt/A///A
Introduction
Scene text appears almost everywhere in daily life and is an essential ingredient for image-text searching [6]. Traditional scene text retrieval 9 typically aims to search images based on word or
phrase queries and could benefit applications such as handwritten signature retrieval [54 [3 and
key frame extraction [35]. However, real-life retrieval needs could be diverse. For instance, people often search for an article with several key words, which cannot be fulfilled with single word. Additionally, non-ocr visual semantics also vital for scene text searching. In this work, we study the multi-query scene text retrieval that aims to handle queries of various types within unified model. This could facilitate applications such as searching disambiguation [i [6] and visual document indexing 48 4.
scene text retrieval methods 9 38 achieve text retrieval by explicitly localizing
the text and matching the query with scene text instances, as shown in Fig. (a). A straightforward
solution the text spotting methods [22 20 49). They first spot the text images and match it using edit distance. However, the two divided stages could cause error accumulation between text spotting and query matching. To mitigate this, detection-based methods [38] 41 (39] represent detected ROIs in dense embedding, integrating text detection and similarity learning into end-to-end framework.
"Correpooniigg author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

walk
Image
image
Localize
Embed
Nordang
Dead Lakes
H/PALS
KULLI
RV'S WELCOME
Cover
Salient Regians
Box- box-based (prior works)
Querry
Query
000
'look and listen'
8%%
Eacode
PVE
Ord florida
20%
Old Fior do
70%
BAILRAD COMOAA STOP
Welcome bezuthiful
18%
Welcome to besutifu
Look AND
Dead lakes
Dead lakes pork
Box-free (ours)
RV's welcome
23%
RV's welcome
85%
(b)
Figure 1: (a) MSTAR achieves soene text retrieval without the aid of box annotations. (b) Image-text matching experiments with VLM [9). Detailed text instances like "welcome to beautiful" and "old florida" in the image receive lower matching scores. While manually covering salient text regions which receive the higher scores, the model can adaptively recognize the detailed text.
Recently, FDP 51 leverages bounding boxes to guide CLIP 30) in focusing text regions to achieve accurate retrieval. Despite these advancements, they require expensive box annotations for training. As different retrieval tasks require varying labels, it is very costly obtain multiple types of bounding boxes, i.e., word-level, text-line, and common object bounding boxes.
Recently, the large-scale box-free pre-training of Vision-Languag Models (VLMs) 18 52 has shown impressive capability on various tasks 555 50). To apply box-free methods for scene text retrieval, we conduct image-text matching experiments, as shown in Fig. (b). The observations reveal that box-free models tend to overlook detailed text instances. Moreover, manually masking salient text regions mitigates this issue by enabling the model to adaptively capture image details. More observations are supplemented in the appendix. Inspired by these, we introduce a box-free model termed Multi-query Scene Text retrieval with Attention Recycling (MSTAR). It starts with leveraging pre-trained VLMs for scene text retrieval. To better capture detailed scene text features, the progressive vision embedding is designed to shift attention from salient regions to insalient areas with less attention iteratively. To support diverse retrieval queries, text queries are encoded by the multi-modal encoder with style-aware instructions. A multi-instance matching module is then designed to establish the cross-modal alignment. In this way, MSTAR seamlessly unifies diverse text queries and aligns image-query embeddings without the need of any positional supervision.
To evaluate the performance of multi-query retrieval, we have carefully built a Multi Multi-Query Text Retrieval (MQTR) benchmark. Beyond traditional word and phrase queries, this dataset introduces two additional, valuable retrieval settings: combined query and semantic query. The combined query comprises several discontinuous key texts (words phrases) to enable more precise retrieval. Semantic queries, on the other hand, are image descriptions that require understanding both scene text and its non-ocr visual context 45). In total, the MQTR dataset includes four styles of queries (word, phrase, combined, and semantic) and 16000 images.
Experiments reveal that existing models struggle to simultaneously handle four types of query on the challenging MQTR dataset. In contrast, our proposed MSTAR performs well in multi-query retrieval. Notably, MSTAR impressively surpasses previous work by average of 8.5% in MAP. Additionally, on seven public retrieval datasets, MSTAR demonstrates competitive performance compared to state- of-the-art box-based methods while significantly reducing annotation costs. Specifically, MSTAR outperforms FDP 51 by 6.4% in MAP on the widely used Total-Text dataset.
We summarize the main contributions as follows: (1) We propose MSTAR, the first box-free method designed for scene text retrieval, which eliminates box annotations and achieves multi-query retrieval. (2) We collect MQTR, comprehensive benchmark with four types of queries and 16,000 images for evaluating multi-query scene text retrieval. (3) Experiments on seven public datasets and the MQTR benchmark demonstrate the advantages of MSTAR.

---

2
Related Work
Scene Text Retrieval Datasets. Existing scene text retrieval datasets primarily focus on single-type retrieval such as word and phrase. The IIIT Scene Text Retrieval dataset [27] is large-scale dataset dedicated to word retrieval comprising 10k images. The COCOText Retrieval dataset is derived from 7k natural images in the COCOText dataset [37]. In addition, several smaller but well-annotated datasets [40] 7 are commonly utilized for evaluating word retrieval performance. The Chinese Street View Text Retrieval dataset 38 contains 23 Chinese queries and 1,667 scene text images from Chinese street views. Besides these word retrieval datasets, the Phrase-level Scene Text Retrieval [51] dataset includes 36 frequently used phrase queries and 1,080 images. The CSVTRv2 (39] dataset supports partial and text-line queries. However, these datasets do not comprehensively support the evaluation of tasks such as combined retrieval and semantic retrieval. In contrast, our MQTR dataset
can support four types of query to satisfy diverse needs in real-world applications.
Scene Text Retrieval Methods. Existing methods generally adopt a paradigm that first localize text regions and then match with the query. In the early attempts, Mishra et al. [27 proposed identifying approximate character locations and indexing words using spatial constraints. More recent approaches try to integrate the two process into an end-to-end trainable framework. Gomez et al. [9] proposed to
combine the detected proposals with the Pyramidal Histogram of Characters [2]. Wang et al. [38] unified the text detector with a cross-modal similarity model into an end-to-end framework. Its updated version (39] proposed the RankMIL and DPMA algorithms to address the partial scene text retrieval problem. Wen el [4] transformed cross- cross-modal similarity into uni-modal similarity using image templates. 7eng et [5] utilized CLIP (30 for scene text retrieval and incorporated box supervision to localize text regions. However, these methods require expensive bounding box annotations for training.
In addition the specifically designed retrieval methods, text spotters can also be applied to retrieval tasks. They first spot text instances from images and then rank with edit distance. Traditional can achieve accurate results. There are also point-supervised (29 and transcription-onk supervised methods (36] 42 (43, which yield limited performance. Above all, the separation of text spotting and query matching often leads to error accumulation, and text spotters struggle to handle multi-query retrieval.
text spotters with boundary supervision 22 20
Unlike above methods, MSTAR achieves text retrieval without the bounding-hox supervision and harmonize various data labels for multi-query scene text retrieval.
3
Method
3.1
Overview
The overall architecture of MSTAR is depicted in Fig. 2 Given a scene text image, the vision encoder extracts image features fa. and the Progressive Vision Embedding module progressively captures the scene text features as vision embeddings Ev. Simultaneously. text queries are encoded as text embeddings Ey by the multi-modal encoder from BLIP-2 (19] with style-aware instructions. The Ey and Er are then fed into the Multi-Instance Matching module to align the cross moos- embeddings optimized with contrastive loss. Additionally, re-ranking process incorporated by inputting both image features and text queries into the multi-modal encoder to compute one-to-one matching scores. During training. MSTAR optimized using both contrastive loss and image-text matching loss. For inference, image-text text pairs are initially ranked by the cosine similarity, and the top K images are further matched with re-ranking process.
3.2 Progressive Vision Embedding
VinguneLanguage Models (VLMs) pre-trained on image-caption pairs often focus more on salient visual concepts such as red circle B3 47]. However, the detailed and subtle scene text instances typically appear various insalient regions of natural scenes. As depicted Fig. (b), the "old florida" and "welcome to beautiful" are overlooked as the model focus on more salient regions. This problem leads to a high miss rate for small text scenarios scene text retrieval tasks. To mitigate this, propose a Progressive Vision Embedding (PVE) approach to extract visual embeddings.
3

---

Progressive Vision Embedding
Image
f
fr
Vision Encoder
SAS
GRAND
M
A
Queries
Co
CT
Ey
Mutli- Mutli-Instane Instance Matching
Multi-modal Encoder (w)
word:
LC
phrase:
TTus
Transfonr formsformer Block o
Block N
Transformer
E
combined:
Linear
mm
semantic:
Ins
Figure 2: Overview of MSTAR. MSTAR is built upon four key components: vision encoder c, the Progressive Vision Embedding (PVB), the multi-modal encoder th and the multi-instance matching module (MIM). PVE incorporates image features fi and the mask M, derived from cross attention map Ce, progressively shifting attention from salient features to fine-grained regions.
Given a scene text image I E Hxxx33 the vision encoder ø encodes into initial image features, denoted as fo RL Then two-layer MLP is used to align the hidden dimensions of & and the multi-modal encoder v stacked with transformer blocks. Subsequently, vb is leveraged to capture the scene text vision embeddings from fo with learnable queries Qi RQxd, The initial vision embeddings are denoted as FQ E Roxd
Since the model tends to focus salient image features (visualized in Fig. 2 capture the detailed text instance representation. To force the model to focus on less salient features, propose the Salient Attention Shift (SAS) module. Motivated by observations in Sec. the SAS uses mask-attention layer to automatically shift image attention. Unlike previous methods [3] that ground truth as supervision, the mask in approach is derived from the cross-attention layers of the multi- modal encoder v. Concretely, we first calculate the mean the cross-ates attention weights of the cross-attention map C.11 Then C. is binarized with binarization algorithm a and inverted to obtain a binary mask M. This is formulated in Eq. [
EV struggles to fully
Ma1
(Cr1),
(1)
where the consists of thresholding with low threshold for coarse filtering, marker-based watershed algorithm for precise binarization, and a connected components algorithm to avoid over- segmentation. In the t-th step, the SAS refines image features follows:
= S(fu1, mask = MM1)),
where S denotes multi-head self-attention, fi1 image features and M,1 is the binary mask. For each pixel M.,, value of indicates that the corresponding image features of the previous iteration received high attention, while a value of 1 indicates lower attention. With M.j, the SAS learns to adaptively reduce the weight of salient features and focus more on the neglected features.
In each iteration, the SAS dynamically renews the image features, illustrated in Fig. 2 Then the multi-modal encoder s embeds the fi to Ey. Then the vision embeddings {EQ, E Sy....E! ET) are concatenated Fl [TT++)Q where the maximum recurrent steps.
3.3 Instruction Aware Text Representation
In unified training of multi quett--qury scene text retrieval, the difference of query styles (e.g., format and characteristics) could cause semantic discrepancy. For example, the phrase query contains several continuous words with linguistic semantics, but the combined query contains several discrete key words. To harmonize the representation of these text queries, short style-aware text instruction is introduced to guide the embedding for each type of query. The process is formulated as follows:
ET (Concate][4) To]),
(3)

---

where s denotes the multi- encoder, T; represents instructions and To denotes text queries. The instructions prompt the & to distinguish query types. As shown in Fig. [ queries of each style are encoded into different representation space during training. To speed up training, all of the text queries (words, phrases, combined, and semantic queries) paired to the image are encoded altogether. The v encodes the queries into text embeddings FT E RXxd which consist of single-word embeddings En E R xd and multi-word embeddings Em E R gNNad, The the total number of text queries paired to the image. The Nw and Nm denote the number of single-word queries and multi-word queries, respectively.
3.4 Multi-Instance Matching
After obtaining the vision embeddings Ev E RT and text embeddings ET E RNd the problem build the one-to-one alignment for the multi-type and multi-instance embeddings. The previous study either aggregate the multiple vision embeddings into one embedding [19] or adopts the late interaction [5 g However, due to the implicit matching mechanism, these strategies need massive training for vision- Mannn-e alignment.
To mitigate these, we propose the Multi-Instance Matching (MIM) module explicitly assign the one-to-one matching relations for vision-language embeddings. MIM comprises two parallel branches for processing single-word and multi-word queries, respectively. In the single-word branch, the Hungarian matching algorithm [17] is exploited to explicitly assign the one-to-one matching relation between Ew E N. xd and By E [[++))xdd Specifically, we first construct a cosine similarity matrix of size Nº x (T+1)Q, Since Nw is typically unequal to (T+I)Q. we pad the matrix with zeros to create a square matrix. Finally, the first No, rows of the results are used for one-to-one correspondences.
In the multi-word branch, since the multi-wond queries contain abundant semantic information, a light-weigh cross-attention layer is used to aggregate the vision features under text constraint in the second branch. This process is formulated as Eq. 4
Ev = F((C(Q Ew, K, == Ev)));
(4)
where F denotes a adaptively shift to cope with different types of queries, i.e., word retrieval relies solely on the first branch, while multi-word queries with the second. Thanks to this flexible alignment approach, mixed training data labels can be effectively leveraged for training multi-query retrieval models.
network and C denotes multi-head cross-attention. The two branches
3.5 Optimization
MSTAR is optimized with both contrastive learning and image-text matching task. Contrastive learning enables the model to separately encode vision embeddings and text embeddings. A dual contrastive loss L. aligns the vision and text embeddings.
Le = aLyy + Lazi
(5)
where a is hyperparameter to maintain the numerical approximation equivalence of the two losses. Since the number of queries is usually greater than the number of images, a is set to 1.5 in our implementation. The image-text matching process simultaneously feeds image features and text queries into the multi-moda encoder zb. An image-text matching score is then computed using linear layer with a two-cell output. This score is optimized using cross-entroe consopy matching loss C.- The overall loss is the sum of L. and Lm.
4 Experiments
4.1 Multi-Query Text Retrieval Dataset
To comprehensively evaluate the performance of models multi-query scene text retrieval, we carefully build the Multi- Query Text Retrieval (MQTR) dataset. The MQTR dataset includes four sub-tasks: word, phrase, combined, and semantic retrieval. The construction of this dataset leverages well-annotated public datasets [37 21
24 51, along with images obtained from Google Image Search. The word, phrase, and combined subsets each contain 5,000 images and the 200 most frequently occurring queries. The semantic subset consists of 1,000 images and 25 queries collected from the web. The semantic subset manually collected with 10- 15 positive images and an equal
1 14
5

---

Dataset
Venue
Word
Phrase
Combined
Semantic
Q. Num
Images
Texal-Text
UDDAR'20
K
K
60
300
CTW
PR'19
X
100
500
IC15
ICDAR'15
V
X
X
100
500
CTR 37
Arxiv' 16
V
X
X
500
7196
STR 9
ECCV' 18
V
X
X
X
50
10k
CSVTR 33
CVPR'21
V
X
23
1667
PSTR [51
ACM MM'24
1080
MQTR
625
16k
Table I: Statistics of public scene text retrieval datasets and our MQTR dataset in terms of query types, number of queries, and number images.
Method
Venue
AVG.
Word
Phrase
Combined
Semantic
Box Based
ABCNet 22
21
24.13
26. 14
15.15
36.47
18.74
MaskTextSpotee
20
ECCV'20
32.43
46.7 46.72
27.53
29.08
26.37
TDSL
CVPR'21
58.25
69.11
40.83
72.71
50.36
Deepsolo
49
CVPR'22
52.04
67.54
25.68
72.14
42.79
TG-Bridge
CVPR 24
54.09
69.89
30.21
75.53
40.73
Box Free
SPTSv2 23
23
35.18
33.56
21.24
50.76
35.16
BLIP-2 19
PMLR'S 23
36.13
17.31
32.76
25.80
68.63
SigLIP
52
CVPR 22
36.06
17.81
32.88
21.81
72.23
BLIP-2 (FT)
19
PMLR'23
58.11
58.09
42.23
60.84
71.24
MSTAR
66.78
73.27
44.22
74.48
75.14
Table Evaluations of MAP% on MQTR. FT denotes fine-tune. The best results are shown in bold.
number of hard negative samples for each query. The hard negatives samples refer to images that contain three types of objects: (1) visual elements with semantics similar to the query (e.g., an apple and the word "apple"), (2) text instances with similar shapes, and (3) text instances with similar meanings. The inclusion of hard negatives poses additional challenges by introducing visually and textually confounding samples, thus assessing the capacity of retrieval models distinguish visual semantics from OCR-based semantics. As demonstrated in Tab. D our MQTR dataset the first comprehensive benchmark support four types of query in scene text retrieval. You can refer the appendix for more construction details and images samples from the datasets.
4.2 Implementation details
The visual encoder d is initialized from ViT-Base-512 of SigLIP 52 initialized from BLIP-2 [9]. The number of query tokens Q1 is set to 64 with interpolation, which consistent with the setting of the vanilla BLIP-2 in our comparison experiments. The weights of the MLP, SAS, and MIM modules are randomly initialized. The MSTAR model was trained on four NVIDIA A800 GPUs and evaluated multi-stage training is adopted with progressive resolution increasing from 512x512, 640x640, to 800x800. For re-ranking, the top 2% of images are selected from the initial retrieval results. For instance, in dataset containing 10k images, only the top 200 images are used for re-ranking.
The multi-modal encoder is
single GPU, using the AdamW optimizer [25]. A
4.3 Multi-Query Scene Text Retrieval
To enable multi-query retrieval, we collect training dataset consisting of 95k images. First, 50k synthetic images with word transcriptions are leveraged from SynthText-900 900k [9]. Then 20k real images containing captions are col- lected from TexiCap (44. We use labels with both image captions and text transcriptions ac- quired with Rosetta
BLIP-2
TDSL 38 SigLIP
FDP
51
MSTAR
19
85.49
89.40
89.56
92.28
95.71
Table 3: Comparisons on Phrase-level Scene Text Retrieval dataset [5]
(3]. In addition, we have

---

Method
Venue
SVT
STR
CTR
Total-Text CTW
IC15
Avg.
FPS
Box Based
Mishra et al.
28
ICCV'13 13
42.70
56.24
0.1
Jaderberg et al.
I6
86.30
66.50
0.3
Gomez et al. 9
ECCV'
83.74
69.83
41.05
43.5
Mafla
PR'21
85.74
71.67
42.2
TDSL
88
CVPR 21
89.38
77.09
66.45
74.75
59.34
77.67
74.16
12.0
Wang et al. 339
TPAMI'24
81.02
72.95
9,3
Wen et al. 41
WSDM'23
90.95
77.40
80.09
11.0
16
51
ACM MM 24
89.63
89.46
79.18
11.8
Box Free
BLIP-2 (FT)
19
PM
88.73
85 ,44
45.75
77.20
82.33
55.13
72.42
37.2
MSTAR
91.31
86,25
60.13
85.55
90.87
81.21
82.56
14.2
MSTAR (+re-rank)
91.11
86.14
65.25
86.96
92.95
82.69
84.18
6.9
Table 4: Comparisons with scene text retrieval methods of MAP% on public word retrieval datasets. The best results are shown bold, and the second results are underlined.
Method
Venue
SVT
STR
CTR
Total-Text CTW
IC15
Avg.
FPS
Box Baxed
ABCNet
TPAMI'21
82,43
67.25
41.25
73.23
74.82
69.28
68.04
17.5
MasteextspoterV3
20
ECCV"20
83.14
74.48
55.54
83.29
80.03
77.00
75.58
2.4
Deepsolo 49
CVPR 23
87.15
76.58
67.22
83.
87.67
82,80"
80.77
10.0
TG-Bridge
|
CVPR'24 24
87.23
81.30
70.08
87.11"
88.39"
83.55
82.94
6.7
Free
SPTSv2 2
TPAMI'23
78.08
62.11
48.39
73.61*
83.30
66.27'
68.63
7.6
MSTAR
91.31
86.25
60.13
85.55
90.87
81.21
82.56
14.2
MSTAR (+re-rank)
91.11
86.
65.25
86.96
92.95
82.69
84.18
6.9
Table 5: Comparisons with mainstream scene text spotting methods. indicates that the model has been fine-tuned on the corresponding training set. The best results are highlighted in bold, and the second results are underlined.
synthesized 25k images with phrase transcrip-
tions using the synthesis engine [10]. Additionally, word or phrase annotations are utilized to form text instance. More training details
nonrepeated combined queries for images containing can be found in the appendix.
On the MQTR dataset, perform evaluation with both the representative box-based models and box-free models for multi-query scene text retrieval. For text spotting methods, we use the normalized edit distance to measure query-iman matching scores following [8). To evaluate the models that can only handle word queries, we calculate the mean similarity of each word as the image-text scores for multi-wor word queries. The codes and weights acquired from their official repositories.
Evaluation on multi-query scene text retrieval. As the results reported in Tab. 2 box-based methods [38 49 I1 typically perform better word queries and combined queries, which demand fine-grained scene text perception. However, these box-based models cannot leverage the rich linguistic semantics for phrase and semantic retrieval. Compared to box-based methods. MSTAR outperforms TG-Bridge IT] by 3.38% and Deepsolo 49 by 5.73% in word retrieval. On the other hand, VLMs such as BLIP-2 (ViT-L) [19] and SigLIP (VIT-B-512) [22] perform better phrase and semantic queries but struggle word and combined retrieval. Compared to them, MSTAR outperforms SigLIP by 11 11.34% phrase retrieval and 2.91 % on semantic retrieval. In terms ofms of overall results, our MSTAR obtains an improvement of 8.53% over previous works on average. These comparison results show the great advantages of our MSTAR multi-query retrieval.
Additionally, MSTAR also evaluated on the phrase-level scene text retrieval dataset As shown in the Tab. 3 our MSTAR achieves 95.71% of MAP on the benchmark.
7

---

Ins
MIM
PVE
CTR
SVT
STR
Total-Text
CTW
IC15
MQTR
X
52.87
90.07
81.57
82.32
87.28
76.71
65.79
X
54.65
90.70
82.81
83.19
88.96
77.15
66,15
X
55.77
91.02
85.00
84.01
90.31
79.23
65.69
60.13
91.31
86.25
85.55
90.87
81.21
66.78
Table 6: Ablation studies on instruction, multi multi-instance matching, progressive vision embedding, denoted as Ins, MIM, PVE, respectively.
4.4 Word-level Scene Text Retrieval
In this part, we present comparison experiments on word-level retrieval. The model is trained on 100k: images randomly sampled from SynthText-9005 [9] and 5k images from MLT-5K (28 dataset. The evaluation setting keeps the same as the previous method [8.. Note that Deepsolo 49 TG-Bridge [1. and SPTSv2 [23] are well fine-tuned on the training set of Texal-Text, CTW and IC15 dataset.
Comparisons with text retrieval methods. We conduct comprehensive evaluations on the test sets of six public datasets, the results presented in Tab. 4 Compared to FDP-RN50x 16 [5T], our MSTAR achieves an improvement of 1.68% on SVT and 6.37% on Total-Text. While MSTAR demonstrates slightly lower performance on the STR dataset, it eliminates the cost of expensive bounding-bos box for training. Compared to TDSL (38. our method significantly outperforms the method across five datasets. Notably, our MSTAR surpasses TDSL by 9.16% on STR, 10.80% on TotalText, and 31.53% on CTW. These results verify the robust capabilities of our model. However, on the CTR dataset that contains extreme small text, our method underperforms TDSL. This is due to the absence of precise box supervision of our method, which is common problem for box-free methods. Overall, MSTAR outperforms TDSL by an average 8.40% in MAP across six datasets. To further enhance performance, we re-rank the top 2% of retrieved images by jointly feeding the text queries and images into the model. This re-ranking strategy improves performance by an additional 1. 1.56% in MAP.
Comparisons with text spotting methods. Tab. 5 shows the comparison results with scene text spotting methods. Compared to box-free method, our model significantly outperforms SPTSv2 with an average improvement of 13.93% MAP. To further verify the advantages of MSTAR, we compare it with saate-o-thh--aa box-based text spotting methods. The results in Tab. 5 indicate that MSTAR achieves competitive performance with the advanced TG-Bridge on average. Moreover, MSTAR offers over twice the inference speed compared to TG-Bridge (14.2 FPS vs. 6.7 FPS) due to the absence of the text detection module. These results show that our model achieves competitive performance with leading fully supervised models while eliminating the bounding box for training.
4.5 Ablation Study
To validate the effectiveness of each component, comprehensive ablation studies were conducted.
Ablation study on three core components. The overall results are reported in Tab. 6 We begin by directly training the vision encoder, MLP and multi-modal encoder with standard contrastive learning. The results suggest that this baseline performs poorly on the fine-grained recognition capability, especially on the CTR and IC15 datasets which features small text. Then instructions are adopted to prompt the model to encode queries which leads to improvement. Subsequently, the MIM is added, which leads to significant improvement of 2. 2.9% on STR and 1.35% on CTW. However, slight performance drop of 0.46% is observed on the MQTR dataset. This may be because of the use of Hungarian matching, which is effective for word-level queries and instance-level alignment, introduces confusion when handling more complex queries.
Lastly, we incorporate PVE which is designed to improve the retrieval performance of the insalient objects such as small and detailed text instances. The results show a substantial improvement on CTR (4.36%) and IC15 (1.98%), validating the effectiveness of PVE in small text scenarios.
A Ablation study on the binary o algorithm. We investigate three variants a, as the results reported in Tab. 7 1) Zero Padding: A binary mask with all zero values is used, which means the mask does not constrain the image attention. The results show only slight improvement on CTR (0.91%), which is probably due to the effectiveness of the progressive representation strategy. 2) The second

---

CTR
Total-Text
IC15
PVE
55.76
84.01
79.23
Zero Pad
56.67
83.79
79.27
TH+CC
59.66
85.17
80.17
TH+WS+CC
60.13
85.55
81.21
Table 7: Ablation studies on a algorithm. TH denotes ThresHolding, CC denotes Connected Components, and WS denotes WaterShed algorithm.
choice of composed of thresholding and connected components. This variant can produce mask to guide the SAS module to refine attention focus. However, it requires repeated tuning of the thresholds. The third choice is to first apply coarse filtering the background with low threshold and then obtain precise binarization results using the watershed algorithm. This approach eliminates the need for complex hyperparameter tuning and achieves substantial improvements, including 4.37% on CTR, 1.54% on Total-Text dataset, and 1.98% on the ICI5 dataset. We adopt the third variant for our model.
Ablation study on the number of iteration steps T in PVE. We investigate the impact of the number of recurrent steps T in PVE. As the re- sults presented in Tab. 8 performance improves significantly as T increases from 0 to 1. As T increases from 1 to 3, we can also observe noticeable improvement on the three datasets. Since the inference speed decreases with the re- current steps increases, we adopt T=1 for the final model to balance efficiency and effectiveness.
T
CTR
Toxal-Text
CTW
FPS
55.76
84.01
90.31
16.5
1
60.13
85.55
90.87
14.2
2
60.47
86.68
90.95
12.9
3
60.87
87.66
91.24
11.2
Table 8: The impact of the recurrent steps
5 Discussion
RAL CLUB
BIG
BIG
(a) 'dream big' tshirt (b) 'lighthouse'
(c) '1888', 'celtic'
(d) 'restaurant'
(c) 'copyright'
Figure 3: Visualization of the text localization of our MSTAR. The image shows the localization of (a) semantic, (b) phrase, and (c) combined query, as well as (d) curved and (e) dense word instances.
Application of MSTAR for text localization. To further validate the effectiveness of MSTAR, use it for text localization using Grad-CAM 332. As illustrated in Fig. 3 MSTAR can localize different types of queries. For example, MSTAR accurately identifies the target text instance "dream big" on a t-shirt, distinguishing it from the "dream big" of the shorts Fig. (a). In addition, MSTAR can also accurately localize curved and dense instances. As Fig. successfully identifies the word "copyright" within a document page image. These results demonstrate that MSTAR can accurately localize text instances without box supervision.
3 (c) shows, MSTAR
Discussion about the pre-trained model. We discuss how the pre-trained model affects performance on the scale of parameters and data. We tested different variants of CLIP [46), BLIP [18, BLIP-2 19], and SigLIP [52) on the CTR dataset. The results show that despite the massive pre-training, the models struggle to achieve retrieval small text instances. Details are provided in the appendix.
Limitations. Although our method shows evident improvements fine- fine-grained scene text retrieval, it still lags behind box-based methods when handling extremely small and dense text instances. This

---

limitation stems from insufficient positional supervision, which is a common limitation to box-free approaches.
Conclusion
This work investigates the fundamental OCR task of scene text retrieval via novel box-free paradigm. incorporates PVE to shift visual attention to fine-grained scene text. Our model demonstrates competitive retrieval performance with state-of-the-ae ducing annotation cost. Moreover, for the first time, we study the multi-query scene text retrieval enabling broader real-world applications. Experiments show that neither box-based methods nor general cross modal retrievers can handle such a challenging task effectively, while our method can serve strong baseline for this task. This study is expected to inspire future research weakly supervised pre-training for visual document foundation models.
box-based methods while significantly re-
Acknowledgements
This work supported by the National Key Research and Development Program of China (Grant 2022YFC3301703 703) and the NSFC (Grants 62206104, 62225603).
References
[1] Aviad Aberdam, David Bensaid, Alona Golts, Roy Ganz, Oren Nuriel, Royee Tichauer, Shai Mazor, and Ron Litman. Clipter: Looking at the bigger picture in scene text recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 21706-21711 2023.
[2] Jon Almazán, Albert Gordo, Alicia Fornés, and Emest Valveny. Word spotting and recognition with embedded attributes. IEEE transactions on pattern analysis and machine intelligence, 36(12);2552-2566, 2014.
[3] Fedor Borisyuk, Albert Gordo, and Viswanath Sivakumar, Rosetta: Large scale system for text detection and recognition in images. In Proceedings of the 24th ACM SIGKDD international conference knowledge discovery & data mining, pages 71-79, 2018.
[4] Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Wiki-llava: Hierarchical retrieval-augmentted generation for multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1818 1826.. 2024.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of IEEE/CVF conference on computer vision and puttern recognition, pages 1290-1299, 2022.
[6] Mengjun Cheng. Yipeng Sun, Longchao Wang. Xiongwei Zhu, Kun Yao, Jie Chen, Guoli Song. Junyu Han, Jingtuo Liu, Errui Ding, et al. Vista: Vision scene text aggregation for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5184-5193, 2022.
[7] Chee- KheeeKKeegg Ch'ng, Chee Seng Chan, and Cheng-Lin Lin Liu. Total-text: toward orientation robustness in scene text detection. International Journal on Document Analysis and Recognition (IJDAR), 23(1):31-52 2020.
[8] Manuel Faysse, Hugues Sibille, Tony Wu, Gautier Viaud, Céline Hudelot, and Pierr Colombo. Colpali: Efficientt document retrieval with vision language models. arXiv preprint arXiv:2407. 4449, 2024.
[9] Lluís Gómez, Andrés Maila, Marçal Rusinol, and Dimosthenis Karatzas. Single shot scene text retrieval. In Proceedings of the European conference on computer vision (ECCV), pages 700-715, 2018.
[10] Ankush Gupta, Andrea Vedaldi, and Andrew 7isserman. Synthetic data for text localisation in natural images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2315-2324, 2016.
[11] Mingxin Huang, Hongliang Li. Yuliang Liu, Xiang Bai, and Lianwen Jin. Bridging the gap between end-to end and two-step text spotting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13688-16118 2024.
10

---

[12] Max Jaderberg, Karen Simonyan. Andrea Vedaldi, and Andrew Zisserman. Reading text in the wild with convolutional neural networks. International journal of computer vision. 116:1-20, 2016.
[13] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez Bigorda. Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. lodar 2013 robust reading competition. In 2013 12th international conference on document analysis and recognition, pages 1484 1493. IEEE, 2013.
[14] Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasckhar, Shijian Lu, et al. Tedar 2015 competition on robust reading. In 2015 13th international conference on document analysis and recognition (ICDAR), pages 1156-1160. IEEE, 2015.
[15] Omar Khattab and Matei Zaharia. Colbert: interaction over bert. Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39-48, 2020.
Efficient and effective passage search via contextualized late
[16] Anastasia Kritharoula, Maria Lymperaiou, and Giorgos Stamou. Large language models and multimodal retrieval for visual word sense disanbiguation. arXiv preprint arXiv:2310.1400 14025, 2023.
[17] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly. (1-2):83-97, 1955.
[18] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- vre-trainig for unified vision- language understanding and generation. In International conference on machine learning, pages 12888 12900. PMLR, 2022.
[19] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman -mage pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[20] Minghui Liao, Guan Pang, Jing Huang, Tal Hassner, and Xiang Bai. Mask textspotter v3: Segmentation proposal network for robust scene text spotting. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 706-722. Springer, 2020.
[21] Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang. Curved scene text detection via transverse and longitudinal sequence connection. Pattern Recognition. 90:337-345, 2019.
[22] Yuliang Liu, Chunhua Shen, Lianwen Jin, Tong He, Peng Chen, Chongyu Liu, and Chen. Abenet v2: Adaptive bezier-curve network for real-time end-to-end text spotting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11))8048-800 8064, 2021.
[23] Yuliang Liu, Jiaxin Zhang, Dezhi Peng, Mingxin Huang, Xinyu Wang, Jingqun Tang, Can Huang, Dahua Lin, Chunhua Shen, Xiang Bai, et Spts v2: single-point scene text spotting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[24] Shanghang Long, Siyang Qin, Dmitry Panteleev, Alessandro Bissacen, Yasuhisa Fujii, and Michalis Raptis. Tedar 2023 competition on hierarchical text detection and recognition. In International Conference on Document Analysis and Recognition, pages 483-497. Springer, 2023.
[25] Loshchilov. Decoupled weight decay regularization. arXiv preprint
2017.
[26] Andrés Mafla, Ruben Tito, Sounak Dey, Lluís Gómez, Marçal Rusinol, Ernest Valveny, and Dimosthenis Karatzas. Real-time lexicon-free scene text retrieval. Pattern Recognition, 100100766, 2021.
[27] Anand Mishra, Karteek Alahari, and CV Jawahar. Image retrieval using textual cues. Tn Proceedings of the IEFE international conference on computer vision, pages 3040-3047, 2013.
[28] Nibal Nayef, Yash Patel, Michal Busta, Pinaki Nath Chowdhury. Dimosthenis Karatzas, Wafa Khlif, Jiri Matas, Umapada Pal, Jean-Christope Burie, Cheng-lin Liu, et al. lcdar2019 robust reading challenge on multi-lingual scene text detection and mroomitioon-mc-ml--220
In 2019 International conference on document analysis and recognition (ICDAR), pages 1582 1587. IEEE, 2019.
[29] Dezhi Peng. Xinyu Wang, Yuliang Liu, Jiaxin Zhang, Mingxin Huang. Songxuan Lai, Jing Li, Shenggao Zhu, Dahua Lin, Chunhua Shen, et al. Spts: single-point text spotting. In Proceedings of the 30th ACM International Conference Multimedia, pages 4272-4281,
11

---

[30] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[31] Brigit Schroeder and Subarna Tripathi. Structured query-hased image retrieval using scene graphs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 178-179, 2020.
[32] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[33] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red cirele? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1 11987-11997, 2023.
[34] Oleksii Sidorov, Ronghang Hu, Mareus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part II 16, pages 742-758. Springer, 2020.
[35] Hao Song, Hongzhen Wang. Shan Huang, Pei Xu, Shen Huang, and Qi Ju. Text siamese network for video textual keyframe detection. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 442-447. IEEE, 2019.
[36] Jingqun Tang, Su Qiao, Benlei Cui, Yuhang Ma, Sheng Zhang, and Dimitrios Kanoulas. You can even annotate text with voice: Transcription-only-suupee International Conference on Multimedia, pages 4164-4163, 2022.
text spotting. In Proceedings of the 30th ACM
[37] Andreas Veit, Tomas Matera, Lukas Neumann. Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07 074400, 2016.
[38] Hao Wang, Xiang Bai, Mingkun Yang, Shenggao Zhu, Jing Wang, and Wenyu Liu. Scene text retrieval via joint text detection and similarity learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4558-4567, 2021.
[39] Hao Wang, Minghui Liao, Zhouyi Xie, Wenyu I Liu, and Xiang Bai. Partial scene text retrieval. IEFF Transactions on Pattern Analysis and Machine Intelligence, 2024.
[40] Kai Wang, Boris Babenko, and Serge Belongie. End-to-end seene text recognition. In 2011 International conference computer vision, pages 4467-146. IEEE, 2011.
[41] Lilong Wen, Yingrong Wang, Dongxiang Zhang, and Gang Chen, Visual matching is enough for scene text retrieval. In Proceedings the Sixteemth ACM International Conference Web Search and Data Mining, pages 447-455 2023.
[42] Lilong Wen, Xiu Tang, and Dongxiang Zhang. Twist: Text-only weakly supervised scene text spotting using pseudo labels. In Proceedings of the 2024 International Conference on Multimedia Retrieval, pages 275-284, 2024.
[43] Jingjing Wu, Zhengyao Fang, Pengyuan Lyu, Chengquan Zhang, Fanglin Chen, Guangming Lu, and Wenjie Pei. Wecromel: Weakly supervised cross-modality contrastive learning for transcription-only supervised text spotting. In European Conference Computer Vision, pages 389-306. Springer, 2025.
[44] Xudong Xie, Ling Fu, Zhifei Zhang, Zhaowen Wang, and Xiang Bai. Toward understanding wordart: Corner-guided transformer for scene text recognition. In European conference computer vision, pages 303-321. Springer. 2022.
[45] Xudong Xie, Liang Yin, Hao Yan, Yang Liu, Jing Ding, Minghui Liao, Yuliang Liu, Wei Chen, and Xiang Bai. Pdf-wukong: A large multimodal model for efficient long pdf reading with end-to-end sparse sampling. arXiv preprint arXiv:2410.05970 2024.
[46] Chuhui Xue, Wenqing Zhang, Yu Hao, Shijian Lu, Philip HS Torr, and Song Bai. Language matters: weakly supervised lasion-language pre-training approach for scene text detection and spotting, In European Conference on Computer Vision, pages 302. Springer, 2022.
[47] Lingfeng Yang. Yueze Wang, Xiang Li. Xinlong Wang, and Jian Yang. Fine-grained visual prompting. Advances in Neural Information Processing Systems, 36, 2024.
12

---

[48] Xiao Yang. Dafang He. Wenyi Huang, Alexander Ororbia, Zihan Zhou. Daniel Kifer, and C Lee Giles. Smart library: Identifying books library shelves using supervised deep learning for text reading. In 2017 ACM/EEE Joint Conference on Digital Libraries (JCDL), pages 1-4, IEEE, 2017.
[49] Maoyuan Ye, Jing Zhang, Shanshan Zhao, Juhua Liu, Tongliang Liu, Bo Du and Dacheng Tao. Deepsolo: Let transformer decoder with explicit points solo text spotting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19348-19357, 2023.
[50] Wenwen Yu, Yuliang Liu, Wei Hua, Deqiang Jiang, Ren, and Xiang Bai. Turning a clip model into a scene text detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6978 6988, 2023.
[51] Gangyan Zeng, Yuan Zhang, Jin Wei, Dongbao Yang, Peng Zhang, Yiwen Gao, Xugong Qin, and Yu Zhou. Focus, distinguish, and prompt: Unleashing clip for efficient and flexible seene text retricval, arXiv preprint 240: 00441, 2024.
[52] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 11975-11986, 2023.
[53] Peirong Zhang, Kai Ding, and Lianwen Jin. Capturing more: Learning multi-domain representations for robust online handwriting verification. arXiv preprint arXiv:2508. 01427, 2025.
[54] Peirong Zhang, Yuliang Liu, Songxuan Lai, Hongliang Li, and Lianwen Jin. Verification With Handwritten Random Digit String. IEEE Transactions on Pottern Analysis and Machine Intelligence (TPAMI), 47(4):3049-300 -004, 2025.
Biometric
[55] Shuai Zhao, Ruijie Quan, Linchao Zhu, and Yang. Clip4str: simple baseline for scene text recognition with pre-trained vision-language model. arXiv preprint arXiv:2300. /4014, 2023.
13

---

Images
Matching scores
90%
Discever
Old flarida
30%
floride
90%
Dead Lakes Park
RV'S WELCOME
Dead lakes pork
Dead lakes park
3.4%
RV's welcome
(a) BLIP-VIT- arge 384
Discover
132
Discover
3.4
Discover
1.56
Discover
-5.03
Old flonida
18.0
Old florida
Old florida
516
florida
81.4
Welcome to beautiful
9.81
Welcome beautiful
21A1
Welcome to beautiful 26.5
->
Welcome beautiful
65.1
Dead lakes park
38.7
Dead lakes park
25.3
Dead lakes park
106
Dead lakes park
58.3
RV's welcome
28.9
RV's welcome
12.1
RV's welcome
58.2
RV's welcome
28.5
(b) SigL
(c) SigL SigLIP-ViT-Large-
Figure 4: Qualitive analysis of VLMs to process in-salient text instances. which is introduced Sec.
(a) BLIP-ViT-Large-384, (b) SigLIP-ViT-Base-512. and (c)
A Further Analysis of Vision Language Models
In this section, we first provide further analysis Vision Language Models (VLM) in processing in-salient text instances introduced in Sec. A.1 Second, we present quantitative evaluation of VLMs on the small-text dataset in Sec.
A.1 Further Observations VLMs
To validate the observations that VLMs tend overlook detailed and insalient text instances, conducted additional experiments with more models. For the BLIP, we calculate the ITM score which calculates the one-to-one matching probability between the image and the query. For the SigLIP models, we calculate the dot products between the vision and text embeddings without normalization following the official code. As illustrated in Fig. (a), the BLIP easily capture the salient text like "dead lakes park" but cannot find the text "old florida". However, it can capture the "old florida" when the salient regions covered. Similar observations presented in Fig. 4 (b) (c).
A.2 Performance of VLMs on Small Text Dataset
To test the pre-trained ability of VLMs on fine-grained perception, we evaluate representative VLMs (CLIP [46], BLIP (18). BL.I--2 [9 SigLIP (52)) the CTR (37) dataset. The CTR dataset is collected from the COCO dataset and features small scene text instances. As the results in Tab. g all of the tested VLMs struggle retrieve images accurately. For instance, the CLIP variants obtain MAP of 8.1 % while the BLIP gets 6.9% of MAP. Moreover, increasing model parameters and seen data does not lead to significant performance gains. Among these models, BLIP-2 has the highest MAP of 13.3% of the tested models, indicating limited capacity of VLMs for small text retrieval.
B MQTR dataset
In this section, we first supplement more construction details and annotation procedures for each subset MQTR dataset in Sec. B.T We then present representative samples from the MQTR dataset in Sec. B.2
14

---

Model
Parameters
Pre-training Data
MAP%
CLIP-RNA
97M
400M images
6.6
CLIP-ViT-BAL
143M
400M Images
6.8
CLIP-ViT-Large
408M
400M Images
8.1
BLIP-ViT-LAI ViT-Large
426M
129M Images
6.9
BLIP-2-V CLIP-2-ViT-Laare Lir--
452M
129M images
13.3
SigLIP-ViTI Sii--ViT--Baa WIT-Base-512
194M
9B Samples
12.8
SigLIP-ViTT-I Large-388
622M
9B Samples
11.7
MSTAR- ViT-Base
270M
60.13
Table 9: Evaluation of the CLIP-style models on the CeCoText dataset. Parameters denotes the parameters of the model.
POS Tag Distribution
Query Length Distribution
Most 20th Frequent Words
Nzuns
30-
Adjectives
enter publications
Verts-
19.r
Prapostions
Percentage (%6
20-
customer
for
Pronouns
Numbers-
15
Adverbs
photo
onnjunctions ons
International
remarks
chou
photography
Others
40
5
10 Frequency
Percentage (%)
per Query
Figure 5: Statistical analysis of the MQTR benchmark.
B.1 Construction details of the MQTR dataset
Word Subset. The word subset includes 5000 images and 200 word queries. The images are sourced from the test set of SVT. CTW, IC15, Total-Text and the CTR. We extract the word-level annotations from datasets and filter out the words with less than characters (e.g. "st"). After that, 200 word queries are selected according to the word frequency.
Phrase Subset. The images of phrase subset include Ik images from PSTR [5]), Ik images manually collected from the Web. Then we use images from HierText (24). The line-level annotations are used and the lines with only one word are filtered out. Lastly, 200 phrase queries are selected according to frequency.
Combined Subset We first all images collected from the CTR and HierText dataset. Given the word and line annotations of an image, we then filter out the queries less than characters. Then implement DFS algorithm compute all the combinations that contain 2-4 words. Top 200 text combinations on the dataset are selected according to frequency and repeat ratios. Then images paired these queries first selected as positive samples. Then images containing words similar to the 200 combined queries are also selected as negative samples according to the edit distance. Lastly, there are 5000 images for the positive samples and negative samples in total.
Semantic Subset The semantic subset is manually collected from the web. we first brainstorm common-used scene text queries and then search for candidate images from the web. In total, the subset consists of 25 queries and 1000 images.
During the selection of multi-word queries (i.e., phrase and combined types), we manually filter out redundant entries. Specifically, queries that are overly repeated semantically similar are removed to enhance diversity in the final set.
15

---

B.2 Visualization of the MQTR dataset
To clarify the characteristies of the MQTR dataset, we show some examples collected from the google image search engine, as presented in Tab. [0] Since scene text typically appears as fine-graine information but the search engine often recommends the most salient images, we collect more images containing queries in fine-grained concepts. For example, "global weekly" is shown as section name of "china daily" newspaper, adding more challenges for models. For example, for the query "dream big", the model may have difficulty distinguishing between "bream big" written the t-shirt and "dream big" written on the shorts.
B.3 Statisties of the MQTR dataset
In total, dataset contains 625 unique queries comprising 1,326 words. As shown in Fig. 5 report statistics including part-of-spece (POS) tag distribution, query length distribution, and the most frequent words, which collectively demonstrate the diversity of our query set across linguistic and structural dimensions.
Query
Examples
CHINAE
CHINADARLY
CHIRADAIKY
CHINADARYY
Positive
newspaper of "global weekly"
CHINADAILY
CHINADAILY
CHIMA A
GAILDAILY
CHINADAILY
CHINADAILY
MEDICAL
Negative
Positive
BIG
HEEP
"dream big" tshirt
BRG
DREAM BIG. IT'S FREE! A
Orenne BI WORK Hande Pray OFTEN
Negative
HARR HADD
8/0
Positive
"I'm ok"
ARE YOU OKAY?
am feeling
IT'S OK
MM FING
Th Good
Negative
fine
Positive
"pizza hut"
A PIZZA
Negative
Table 10: Examples from the MQTR dataset. The Positive represents the GT images. The Negative denotes hard negative sample described Sec.
16

---

Experiment Details and Visualization Analysis
C.1 Computational Efficiency Analysis.
Thanks to the design of the PVE module, image features are fed into the ViT only once. Subsequent iterations involve merely the SAS module and the Multi-modal Encoder, both of which lightweight and thus enable good trade-off' between accuracy and efficiency. We analyze the computational cost and latency single A800 GPU, where the Baseline removes the PVE module from MSTAR.
Method GFLOPs Latency (s) FPS CTR
Baseline 248
0.0606
16.5 55.76
MSTAR 310
0.0704
14.2 60.13
Table 11: Computation and efficiency comparison on a single A800 GPU.
MSTAR improves accuracy by 4.37% on the challenging CTR dataset, while achieving an inference speed of 14.2 FPS, over twice as fast as TG-Bridge (6.7 FPS), and maintaining comparable performance across all six benchmarks.
As shown in Table 11
C.2 Text Region Localization Analysis.
We further evaluate the localization accuracy of the predicted text regions. Binary ground-truth (GT) masks are constructed by extracting the polygon coordinates of all text regions from the CTW dataset. For comparison, we also derive binary masks from the gross-attention maps of BLIP-2 using the same processing pipeline ours. As reported in Tab. 12 we compute the Intersection over Union (IoU) between each predicted mask and corresponding GT mask, and report both the mean IoU and the number of high-quality masks with loU 2 0.5 across 500 images.
Method Mean IoU High-Quality Masks (IoU > 0.5)
BLIP-2 21.78
129 (25.8%)
MSTAR 50.82
304 (60.8 ))
Table 12: Quantitative comparison of text region localization on the CTW dataset. MSTAR produces substantially more accurate and higher-quality text masks than BLIP-2.
C.3 Scale-wise Evaluation on ICDAR2015.
We further evaluate MSTAR the ICDAR2015 dataset by analyzing performance across different text scales. For each text instance, we compute the ratio between its bounding- area and the image area using annotations from the original datasets, and group instances into scale intervals shown in Tab
For each interval, we calculate the AP score for queries whose corresponding text instances fall within the specified scale range. For instance, given the query "apple" and a target scale range ((a, b]]. we exclude all images where "apple" appears but its area ratio is not within that range. The AP score then computed using the remaining valid images.
C.4 Training Details
To ensure the reproducibility of our model, we provide the training process and the training hyper parameters, which are reported in Tab. 13 We adopt a progressive training recipe. In the first state, the model is trained on both synthetic data Dayn and real data D at an image resolution of 512. We use learning rate of le-5 and linear cosine schedule with 100 warm-up steps. In this stage, we use two-layer MLP to align the visual encoder and multi-modaa encoder. In the second stage, our model is fine-tuned at the resolution of 640 only on the real data Dreat- For simplicity, the hyper parameters keep the same with the first stage. In the third stage, we fine-tune the model at higher resolution of 800 on the real data D In the forth stage, the visual encoder freezed and only the MLP, modal encoder, MIM, and SAS module optimized at resolution of 800. For
17

---

Figure 6: Scale-wise performance comparison on ICDAR2015.
Phase
Phase 2
Phase 3
Phrase
Image Resolution
512
640
800
800
Learning Rate
le-5
le-5
5e-6
5e-6
Warm-Up steps
100
100
0
Freeze ViT
False
False
False
True
Precision of VIT
Float
Query of ab
64
RandomCrop
True
Dataset
(Dayn: Dreat }
Dreal
Dreat
Dreal
Table 13: Training details of our model.
the word retrieval experiment, the synthetic data D refers to 100K images randomly sampled from SynthText-9000 and the real data Dreal comes from MLTT-K. For the multi-query experiment, Dsyn includes 50K images randomly sampled from Synth Text-900k and 25k images with phrase transcriptions with the synthesis engine. Dieal is the training set from the TextCap dataset. The labels are the image captions and text transcriptions acquired with Rosetta.
C.5 Experimental Visualization Analysis
We present qualitative analysis of the retrieval results. As illustrated in Tab. 14 our method can not only effectively leverage linguistic priors for phrase and semantic queries, but also perceive fine-grained scene text instances achieve word and combined retrieval. For example, MSTAR successfully retrieves the combined query like "reserved" and "some" even when they appear as subtle watermarks in pictures, as depicted in the last row.
18

---

Query
Retrieval results
SPEED
SPEED LIMIT 35
Bridge II
30
LIMIT 35
3250 FINE MINIMUM
"speed limit 25"
PEED LIMIT 25
specD 25
MSTAR
A Atlnnicc
FUr 5L ON ZONE
SLOW ZONE
Bridge
1I
"hydrate or diedrate" is written water bottle
MSTAR
50%
STAB 176
GARDA BMN CLUB MUSL SOLUROSI
VILLAGE GROWERS MARKET
BLIP-2 (FT)
H.HEWS&
19
MAY 12TH 2002 CHARRTY 1O MALIN READ
No. CA.RIDBRDDG
"may"
MASS
Wide MUAIE 1012
CARDA/ BMN CLUB
PARADISE LIONS SATURDAY FARMERS Market
CARTERS SEBAM FAIR
MSTAR
TUPLE MOLAROOIE HERAD
VICTORIA PARK
MAY 1TT 2007
MAY 2511 28th
D0P
BLIP-2 (FT)
119
"reserved". "some"
MSTAR
Table 14: Qualitive analysis of the retrieval results. The green boxes mean correct samples and the red boxes denote false images. The queries and images are sampled from the MQTR dataset and CTW 21 dataset.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction are presented based on the core contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Sec. 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
The theoretical result are clearly clarified.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We clearly describe the method in Sec. 3 and include the implementation
details in Sec. [4.2, evaluation details in Sec. 4.3 and Sec. [4.4] Moreover, we supplement more training details in the appendix. Lastly, we also offer our code and data to ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will provide the code, weights and data in the supplementary files.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wtipss.ccppbllcc//.ececccccccccc guwdes/ddeeeee CooeeSumeeeeeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We include the implementation details in Sec. 4.2 evaluation details in Sec. 4.3 and Sec. Moreover, we supplement more training details the appendix. C.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Sure. The evaluation criteria keeps the same with previous studies, which is mentioned in Sec. 4 We conducted extensive experiments on seven widely-used public
datasets. These experimental results are of statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include this in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tte:://neerips/e wwwperpp..cpppllc//eeeeeeeeeeeee cccppullic/thiiccciidel........
Answer: [Yes]
Justification:
The research is fully conformed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have claimed the impact of scene text retrieval Sec. D while this research is step forward to bring new box-free paradigm to address the task.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The model and dataset is commonly used for OCR tasks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the papers and will respect the licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Do not involve.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This research does not involve LLMs any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hbtpe://heurpss.ccooooorrrrre//p for what should or should not be described.
26


Reviews:
[{"review_id": "5E9RnXWmlJ", "reviewer": "Reviewer_Qggy", "summary": "This work proposes a box-free scene text retrieval method that eliminates the need for bounding box annotations. It employs progressive vision embedding and style-aware queries to enhance text representation and vision-language alignment. Additionally, it introduces MQTR, a new dataset for multi-query retrieval, on which the method outperforms existing approaches.", "questions": "1. Regarding the new MQTR dataset, the selection of 200 queries for the word, phrase, and combined subsets was based on frequency from a collection of existing datasets. How did you ensure these queries are diverse and representative, and not biased towards the specific text distributions of the source datasets (e.g., SynthText, TextCap)?\n2. The paper identifies difficulty with \"extremely small text\" as a primary limitation. Why donot your method deal with this limitation?\n3. How to set proper recurrent steps T?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This paper proposes a new box-free method specifically designed for multi-query scene text retrieval. This work may benifit ***\n2. The new dataset MQTR dataset enables a more thorough and realistic evaluation of retrieval models and will likely spur further research in this domain.\n3. MSTAR demonstrates impressive performance gains across seven public datasets and the new MQTR benchmark.\n\n\nWeaknesses:\n1. The model's performance deteriorates on datasets with extremely small and dense text, where box-based methods still hold an advantage.\n2. Its core attention recycling mechanism introduces a direct trade-off between higher retrieval accuracy and slower processing speed.\n3. The algorithm for shifting the model's attention is complex, potentially posing a challenge for replication and future development.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful feedback on our work! Because both W1 and Q2 ask about the \"extremely small text\", we answer the two questions together.\n\n---\n\n***W1: The model's performance deteriorates on datasets with extremely small and dense text, where box-based methods still hold an advantage.***\n\n***Q2: The paper identifies difficulty with \"extremely small text\" as a primary limitation. Why does your method not deal with this limitation?***\n\nThank you for the valuable comment. Handling extremely small text remains inherently challenging for all box-free methods without precise box supervision. Nonetheless, our work takes a concrete step forward within the box-free paradigm. For instance, on the CTR dataset, which features small and complex text layouts, our model achieves a MAP of 60.13%, clearly outperforming prior methods such as SPTSv2 (48.39%) and BLIP-2 (45.75%). \n\nWhile extremely small text remains an open problem for box-free methods, our method brings unique advantages: it not only eliminates the box annotation cost, but also broadens the multi-query scene text retrieval applications. \n\nTherefore, we see this as a meaningful advancement in addressing fine-grained visual text understanding within box-free methods.\n\n---\n\n***W2: Its core attention recycling mechanism introduces a direct trade-off between higher retrieval accuracy and slower processing speed.***\n\nThank you for the comment. Due to the design of our PVE module, the image features are only fed into the ViT once. Subsequent iterations involve only the SAS module and the Multi-modal Encoder. These two components are relatively lightweight in terms of computation time, enabling our method to achieve a favorable balance between accuracy and efficiency.\n\nTo study the computation costs of each component, we provide the analysis of the computation and latency of MSTAR. The baseline refers to the results obtained by removing the PVE module from MSTAR. The experiments are conducted on a single A800 GPU.\n\n| Method   | Computation (GFLOPs) | Latency (s) | FPS  | CTR    |\n|----------|----------------------|-------------|------|--------|\n| Baseline | 248                  | 0.0606      | 16.5 | 55.76  |\n| MSTAR    | 310                  | 0.0704      | 14.2 | 60.13  |\n\nAs the results show, our method improves precision by 4.37% on the challenging CTR dataset. It also achieves an inference speed of 14.2 FPS, which is more than twice as fast as TG-Bridge (6.7 FPS), while maintaining comparable performance across the six datasets.\n\n---\n\n***W3: The algorithm for shifting the model's attention is complex, potentially posing a challenge for replication and future development.***\n\nThank you for the comment. In our design, the PVE module performs attention shifting in an effective (Reviewer 5CKX) and interesting (Reviewer EPGg) manner. It directly reuses the cross-attention weights from the model as binary mask to shift the focus toward less salient features. Notably, our approach eliminates the complex pipeline adopted by previous box-based methods, which typically require explicit scene text detection [1] and ROI-based embedding [2]. This further reducing implementation complexity.\n\nRegarding future development, all components in our design are modular and plug-and-play. The attention shift process can be iteratively controlled via a hyperparameter T. They can be flexibly integrated with different types of text queries and various visual backbones. We believe this work can inspire further research and benefit the community.\n\nRegarding reproducibility, we have included all training details, code, and pretrained weights in the supplementary material. We will release them publicly to the open-source community. These resources ensure the reproducibility of our results.\n\n[1] Huang, Mingxin, et al. \"Bridging the gap between end-to-end and two-step text spotting.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Wang, Hao, et al. \"Partial scene text retrieval.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).\n\n---\n\n***Q1: Regarding the new MQTR dataset, the selection of 200 queries for the word, phrase, and combined subsets was based on frequency from a collection of existing datasets. How did you ensure these queries are diverse and representative, and not biased towards the specific text distributions of the source datasets (e.g., SynthText, TextCap)?***\n\nWe demonstrate the diversity and representativeness of our queries from three perspectives:\n\nFirst, our source data is collected from 8 widely used datasets, which enables diversity of both queries and images. These datasets cover a wide range of text types, including single-word text, curved text (CTW), line-level text (CTW, HierText), small text (CTR), and dense text (HierText), among others. In addition, the images and annotations used in MQTR are entirely independent of our training datasets, including SynthText and TextCap.\n\nSecond, during the selection process for multi-word queries (i.e., phrase and combined types), we manually filtered out entries with high redundancy. For example, queries that were overly repeated or semantically similar were removed to further ensure diversity in the final query set.\n\nThird, we provide a detailed statistical analysis of the queries. In total, there are 625 unique queries comprising 1,326 words. We report statistics including part-of-speech (POS) tag distribution, query length distribution, character frequency, and word frequency. These analyses demonstrate the diversity of our queries across multiple linguistic and structural dimensions.\n\n1. POS Tag Distribution\n\n| POS          | Percentage (%) |\n| ------------ | -------------- |\n| Nouns        | 50.45          |\n| Adjectives   | 17.65          |\n| Verbs        | 13.35          |\n| Prepositions | 5.51           |\n| Determiners  | 3.54           |\n| Pronouns     | 2.64           |\n| Numbers      | 2.49           |\n| Adverbs      | 2.49           |\n| Conjunctions | 0.53           |\n| Modals       | 0.38           |\n| Others       | 0.98           |\n\n2. Query Length Distribution\n\n| Words per Query | Percentage (%) |\n| --------------- | -------------- |\n| 1 word          | 32.5           |\n| 2 words         | 24.2           |\n| 3 words         | 20.6           |\n| 4 words         | 5.4            |\n| 5 words         | 12.5           |\n| 6 words         | 2.1            |\n| 7 words         | 1.5            |\n| 8 words         | 0.8            |\n| 9 words         | 0.5            |\n\n3. Most Frequent Words (Top 20):\n\n```\nsan (23), diego (18), the (18), references (16), publications (15), \nyour (15), a (14), customer (13), for (12), to (12), \nservice (11), search (11), you (10), photo (10), site (10), \ninternational (9), remarks (9), chau (9), photography (8), new (8)\n```\n---\n\n***Q3: How to set proper recurrent steps T?***\n\nThe choice of T should balance effectiveness and efficiency. We report the ablation results of T in Table 8. As T increases from 0 to 1, the performance improves significantly, i.e., a 4.37% improvement on the CTR dataset. Further increases from T = 1 to T = 3 continue to yield consistent gains across all three datasets. However, these gains come at the cost of reduced inference speed, as the number of recurrent steps increases. To balance accuracy and efficiency, we adopt T = 1 as the final configuration. This setting provides the best trade-off, achieving notable performance improvements while maintaining practical inference speed."}, {"review_id": "RsqkxMNW1j", "reviewer": "Reviewer_EPGg", "summary": "A box-free scene-text retreival method for multiple query styles is proposed in this manuscript. To achieve box-free retreival, this method utilizes saliency map from vision encoder for text spotting. An attetion recycling mechanism is proposed in this method for saliency attention shift in order to capture the text from saliency part to less saliency part. An instruction aware text representation is introduced for retrieval with free-style query. The instruction indicates the query type, and queries of different type are mapped to their own feature space during trianing. A multi-instance matching method is involved to match vision features under different attention shift with text queries. The proposed method is evaluated on multiple benchmark datasets and a newly collected dataset for free-style scene-text retreival task. The experiment results show the effectiveness of the proposed method.", "questions": "1.\tIn Eq.2, is the mask directly applied on the feature map or it needs a mapping or it is an attention map?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n \n1.\tThe method achieves performance comparable to the state-of-the-arts not only on scene text retrieval task but also on scene text spotting task;\n\n2.\tThe introducing of attention shift for captureing scene-text of different scales is interesting, and help achieve good performance;\n\n3.\tA new dataset is collected for free-style scene text retreival task;\n\n4.\tThe propsoed method is extensively evaluated;\n\n**Weakness:**\n\n1.\tThe authors claim that the proposed method is able to process text queries of different styles, and an instruction aware text representation is proposed to well process the text queries of four styles. This instruction-based way may not be pratical in real world application as the users tend to input queries without indicating their styles. An addiional query style recognition will make this work more complete.\n\n2.\tVision Encoder like Sig-LIP may not be suitable for scene text saliency detection as the model is pre-trained on data most of which do not have scene text in image. The saliency may appear at the apearance of the object the corresponding text describe, the appeat at the scene text. The finetuning needs large scale of data to calibrate the vision encoder for scene text saliency adjustment;\n\n3.\tThe authors state that the attention recycling can help discover scene text with less saliency. However, an evaluation specifically designed for small-scale scene text is not conducted. It would be great to also include this experiment to make the evaluation complete. The evaluation of scene text retreival performance under different text scale is necessary;\n\n4.\tThe attention shift (or progressive enhancement of feature) is a popular strategy in computer vision, which has been used in other vision task such as object saliency detection [1], action recognition [2], action prediction [3], image captioning [4]. It would be great to also review these works in related work;\n\n5.\tTypos:\n\n* In line 153, the sentence “The N is the total number of text queries paired to the image.” is repeated twice;\n* In the caption of Table 3. The word “phrase-level”  is wrongly written as “Prase-level”;\n\n[1] Siris, Avishek, Jianbo Jiao, Gary KL Tam, Xianghua Xie, and Rynson WH Lau. \"Inferring attention shift ranks of objects for image saliency.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12133-12143. 2020.\n\n[2] Weng, Junwu, Donghao Luo, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, Xudong Jiang, and Junsong Yuan. \"Temporal distinct representation learning for action recognition.\" In European Conference on Computer Vision, pp. 363-378. Cham: Springer International Publishing, 2020.\n\n[3] Stergiou, Alexandros, and Dima Damen. \"The wisdom of crowds: Temporal progressive attention for early action prediction.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14709-14719. 2023.\n\n[4] Pedersoli, Marco, Thomas Lucas, Cordelia Schmid, and Jakob Verbeek. \"Areas of attention for image captioning.\" In Proceedings of the IEEE international conference on computer vision, pp. 1242-1250. 2017.", "was_revised": true, "final_justification": "My concerns have been well-addressed. The idea of progressive attention can not only be applied in small-scale text retrieval, but be extended to discover small-scale objects as well. Therefore the idea proposed in this paper is valuable. After reading the rebuttal and the reviews from other reviewers, I tend to keep my rating as borderline accept.", "rebuttal": "Thank you very much for your thoughtful feedback on our work! \n\n---\n\n***W1: The authors claim that the proposed method is able to process text queries of different styles, and an instruction aware text representation is proposed to well process the text queries of four styles. This instruction-based way may not be pratical in real world application as the users tend to input queries without indicating their styles. An addiional query style recognition will make this work more complete.***\n\nThanks for your valuable suggestion! To enhance the practicality of our method in real-world scenarios where users often do not explicitly indicate query styles, we introduced a rule-based query style classification module. \n\nSince OCR text and general visual concepts can be confusing, users are asked to put OCR-related text inside single quotes. First, we apply NLTK tokenizer to segment the input query into tokens. Based on the tokenized results and simple pattern matching, we classify queries into four categories as follows:\n\n- **Word queries:** The query consists of a single quoted token, e.g., `'apple'`. When the tokenized query contains exactly one word enclosed in single quotes, it is recognized as a word query.\n- **Phrase queries:** The query is a single quoted sequence of multiple continuous words forming a meaningful phrase, e.g., `'scene text retrieval'`. If the quoted text contains more than one word without any spaces separating different quoted segments, it is identified as a phrase query. \n- **Combined queries:** The query contains multiple quoted words or phrases separated by spaces, e.g., `'retrieval' 'OCR'`. If the input has multiple quoted segments separated by spaces, it is classified as a combined query.\n- **Semantic queries:** Queries that have a more free-form structure, possibly mixing quoted and unquoted parts, such as `bottle saying 'purified drinking water'`. These are identified when the query does not strictly follow the above patterns.\n\nThis approach leverages simple regular expressions over tokenized input, providing a lightweight and robust classification mechanism without dependence on heavy NLP models. We believe this addition improves the completeness and applicability of our framework significantly.\n\n---\n\n***W2: Vision Encoder like Sig-LIP may not be suitable for scene text saliency detection as the model is pre-trained on data most of which do not have scene text in image. The saliency may appear at the apearance of the object the corresponding text describe, the appeat at the scene text. The finetuning needs large scale of data to calibrate the vision encoder for scene text saliency adjustment;***\n\nThanks for your valuable suggestion! The reasons for choosing SigLIP as the vision encoder are two-fold:\n\nFirst, vision encoders such as SigLIP exhibit strong vision-language alignment capabilities, as they are pretrained on massive web image–caption pairs, which often include rich scene text information [1,2].\n\nSecond, SigLIP and similar encoders demonstrate impressive zero-shot performance on scene text understanding tasks. For instance, TCM [3]/TCM++ [4] shows that neurons in CLIP are inherently responsive to textual signals, and CLIP can enhance text spotting models with significantly reduced supervision. Additionally, FDP [5] reports that CLIP achieves strong zero-shot performance on the SVT dataset, with an accuracy of 65.07%, further validating its effectiveness in handling scene text without task-specific fine-tuning.\n\n[1] Fan, David, et al. \"Scaling language-free visual representation learning.\" arXiv preprint arXiv:2504.01017 (2025).\n\n[2] Cao, Liangliang, et al. \"Less is more: Removing text-regions improves clip training efficiency and robustness.\" arXiv preprint arXiv:2305.05095 (2023).\n\n[3] Yu, Wenwen, et al. \"Turning a clip model into a scene text detector.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n[4] Yu, Wenwen, et al. \"Turning a clip model into a scene text spotter.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 46.9 (2024): 6040-6054.\n\n[5] Zeng, Gangyan, et al. \"Focus, distinguish, and prompt: Unleashing clip for efficient and flexible scene text retrieval.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n---\n\n***W3: The authors state that the attention recycling can help discover scene text with less saliency. However, an evaluation specifically designed for small-scale scene text is not conducted. It would be great to also include this experiment to make the evaluation complete. The evaluation of scene text retreival performance under different text scale is necessary;***\n\nWe conduct evaluation experiments on the ICDAR2015 dataset by analyzing performance across different text scales. Specifically, we compute the ratio of each text instance’s area to the corresponding image area based on the dataset annotations, and then group these instances into scale intervals as shown in the table below.\n\nTo evaluate retrieval performance at different scales, we calculate the AP score for each query using only the images that contain matching text instances falling within the specified scale range. For example, given a query \"apple\" and a target scale range \\((a, b]\\), we discard any images where the word \"apple\" appears but its occupied area ratio is not within \\((a, b]\\). The AP score is then computed over the remaining valid images.\n\n\n| **Scale Range** | **Baseline** | **+PVE** | **Δ**  |\n| --------------- | ------------ | -------- | ------ |\n| (0.01, 0.1]     | 81.35        | 82.59    | +1.24% |\n| (0.005, 0.01]   | 78.34        | 80.01    | +1.67% |\n| (0.0025, 0.005] | 78.75        | 81.02    | +2.27% |\n| (0.001, 0.0025] | 76.47        | 79.38    | +2.91% |\n| (0.0005, 0.001] | 65.23        | 69.25    | +4.02% |\n| (0, 1)          | 79.23        | 81.21    | +1.98% |\n\nThe first column indicates the text scale intervals, the second shows the baseline performance without PVE, the third presents the results with our proposed PVE module, and the last column reports the performance gain. As shown, the smaller the text instance, the more significant the improvement brought by PVE. This confirms that our method is particularly effective at capturing fine-grained text, leading to larger gains in challenging small-scale scenarios.\n\n\n---\n\n***W4: The attention shift (or progressive enhancement of feature) is a popular strategy in computer vision, which has been used in other vision task such as object saliency detection [1], action recognition [2], action prediction [3], image captioning [4]. It would be great to also review these works in related work***\n\nThanks for the suggestions, we will add these related works in the revised versions.\n\n---\n\n***W5: Typos: In line 153, the sentence “The N is the total number of text queries paired to the image.” is repeated twice; In the caption of Table 3. The word “phrase-level” is wrongly written as “Prase-level”;***\n\nThanks for pointing this out. We will correct the repeated sentence in line 153 and the typo “Prase-level” in the caption of Table 3 in the revised version.\n\n---\n\n***Q1: In Eq.2, is the mask directly applied on the feature map or it needs a mapping or it is an attention map?***\n\nThe mask is directly add to the attention weights like Mask2Former. Here's a concise explanation of Eq.2.\n\nGiven the input image features $f_{t-1} \\in \\mathbb{R}^{B \\times L \\times D}$, the model computes:\n\n$$\nQ = f_{t-1}W_Q,\\quad K = f_{t-1}W_K,\\quad V = f_{t-1}W_V,\n$$\n\nwhere $Q, K, V \\in \\mathbb{R}^{B \\times L \\times D}$ are the projected query, key, and value embeddings. Then, the self-attention scores are calculated by:\n\n$$\n\\text{Scores} = \\frac{QK^\\top}{\\sqrt{d}}.\n$$\n\nTo suppress salient (already attended) regions and enhance less-attended areas, we apply the **binary mask** **$M_{t-1} \\in \\mathbb{R}^{B \\times L}$**, which is derived from the inverted cross-attention map. This mask is broadcast and added to the attention scores:\n\n$$\n\\text{Scores} = \\text{Scores} + M_{\\text{expanded}},\n$$\n\nwhere masked positions (i.e., previously over-attended regions) receive large negative values, effectively reducing their contribution via:\n\n$$\n\\text{Probs} = \\text{softmax}(\\text{Scores}).\n$$\n\nFinally, the new features are computed as:\n\n$$\nf_t = \\text{Probs} \\cdot V.\n$$\n\nThrough this masked attention, the SAS module adaptively shifts focus away from dominant regions and encourages the model to attend to previously overlooked text instances.\n\n[1] Cheng, Bowen, et al. \"Masked-attention mask transformer for universal image segmentation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022."}, {"review_id": "uuXigIxuc5", "reviewer": "Reviewer_5CKX", "summary": "The paper studies the problem of scene text retrieval. To this end, the authors propose a new method named MSTAR, which eliminates bbox annotations and can handle multi-query retrieval. The proposed method consists of several components: Progressive Vision Embedding, Instruction Aware Text Representation and Muti-Instance Matching. The key insight is the Progressive Vision Embedding, which gradually shifts attention from salient features to fine-grained regions, via applying the masks derived from the cross attention map. In addition to the method, the authors have also collected a new dataset, MQTR, for the evaluation of multi-query retrieval. Experiments on existing benchmarks as well as the newly proposed benchmark show the proposed method generally achieves a better performance. Ablation study validates the effectiveness of each proposed module.", "questions": "Given the weaknesses of the paper, especially the significance of the problem the paper studies, I am more inclined to reject the paper.\n\nI would suggest the authors to answer the 3 questions and concerns as in the weakness section.", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Dataset contribution. Compared with existing datasets, the newly constructed dataset has more images and queries, and most importantly, supports all Word, Phrase, Combined and Semantic.\n\n2. Method contribution. The idea of Progressive Vision Embedding makes sense to me. It is an effective way of getting rid of the requirement of bbox.\n\nWeaknesses:\n\n1. Significance of the problem. The problem of scene text retrieval is a very specific and narrow task in the retrieval community, and may not be of interest for a large audience. \n\n2. Performance decrease for re-ranking. In Table 5, for MSTAR, the performance degrades after re-ranking. The authors should give a reasonable explanation.\n\n3. Paper writing. There are several writing issues: On the title, should be 'Box-Free Multi-Query'; In Line 106, should be 're-ranking'; In Line 104, should be 'BLIP-2'; In Table 5 caption, should be 'fine-tuned'. The paper needs further proofreading and attention to make the wordings and spellings correct and professional.", "was_revised": true, "final_justification": "Both of my W1 and W2 still remain unresolved (see my comment for details). Therefore, I am keeping my original rating.", "rebuttal": "Thank you very much for your thoughtful feedback on our work! \n\n---\n\n***W1: Significance of the problem. The problem of scene text retrieval is a very specific and narrow task in the retrieval community, and may not be of interest for a large audience.***\n\nWe would like to clarify that scene text retrieval is a fundamental task in the fields of document analysis and OCR. It has attracted significant attention from the research community, with a number of influential works published in top-tier venues such as TPAMI [1], CVPR [2,7], ECCV [5], ICML [8], and ACM MM [4]. \n\nMoreover, scene text serves as an important cue in information retrieval. Incorporating scene text information has been shown to improve the performance of general image-text matching tasks on benchmarks such as MS-COCO and Flickr30K [3,6]. \n\nFinally, due to the inherently fine-grained and intricate nature of scene text, we believe it presents unique challenges that warrant further investigation. Our work proposes a meaningful box-free method for unified multi-query scene text retrieval (Reviewer w1db). The PVE is well-motivated and interesting for capturing scene text at different scales (Reviewer EPGg, w1db). The newly introduced MQTR dataset enables thorough and realistic evaluation of retrieval models (Reviewer Qggy, EPGg). To sum up, we argue that scene text retrieval is not only necessary but also a valuable direction for the NeurIPS community.\n\n[1] Wang, Hao, et al. \"Partial scene text retrieval.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).\n\n[2] Wang, Hao, et al. \"Scene text retrieval via joint text detection and similarity learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[3] Cheng, Mengjun, et al. \"Vista: Vision and scene text aggregation for cross-modal retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[4] Zeng, Gangyan, et al. \"Focus, distinguish, and prompt: Unleashing clip for efficient and flexible scene text retrieval.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n[5] Gómez, Lluís, et al. \"Single shot scene text retrieval.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n\n[6] Mafla, Andrés, et al. \"Stacmr: Scene-text aware cross-modal retrieval.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021.\n\n[7] Qin, Xugong, et al. \"CLIP is Almost All You Need: Towards Parameter-Efficient Scene Text Retrieval without OCR.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[8] Li, Gengluo, Huawen Shen, and Yu Zhou. \"Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts.\" arXiv preprint arXiv:2506.04999 (2025).\n\n---\n\n\n***W2: Performance decrease for re-ranking. In Table 5, for MSTAR, the performance degrades after re-ranking. The authors should give a reasonable explanation.***\n\nIn Table 5, we observe the performance improves by 1.62% on average after re-ranking. However, the results slightly degrade on the SVT (91.31% vs 91.11%) and STR (86.25% vs 86.14%) datasets. This is because the re-ranking is an additional component to refine the top-k results. Re-ranking can only reorder a fixed set of top-k results. If the top-k set already contains false positives (e.g., visually similar but semantically irrelevant items), the reranker may overfit to superficial cues (e.g., layout or text frequency) and push wrong candidates higher, thereby amplifying early-stage errors.\n\n---\n\n***W3: Paper writing. There are several writing issues: On the title, should be 'Box-Free Multi-Query'; In Line 106, should be 're-ranking'; In Line 104, should be 'BLIP-2'; In Table 5 caption, should be 'fine-tuned'. The paper needs further proofreading and attention to make the wordings and spellings correct and professional.***\n\nThank you for pointing this out! We will correct all the mentioned issues and perform thorough proofreading to ensure that the paper meets the professional standards in wording, spelling, and overall clarity."}, {"review_id": "G8qnbf2o8F", "reviewer": "Reviewer_w1db", "summary": "The paper introduces MSTAR, a box-free approach for multi-query scene text retrieval, eliminating the need for costly bounding box annotations during training.  They propose Progressive Vision Embedding to Dynamically captures multi-grained text representations by shifting attention from salient to insalient regions, and introduce Multi-Instance Matching (MIM) to enhance cross-modal alignment between vision and text embeddings. Additionally, they constract a new benchmark named MQTR with 16K images and four query types to evaluate multi-query retrieval.", "questions": "1.In Fig. 1 b, while it's understandable that VLMs tend to overlook detailed text instances, why do they still maintain high attention scores for regions after covering the salient text areas? I.e., after masking \"Dead lakes park,\" the attention score remains as high as 96%.\n2.In Tab. 8, why choose T=1 as the final configuration? Table 8 shows better performance at T=3, but authors chose T=1 for efficiency reasons - is this trade-off reasonable?\n3.Can you provide more quantitative analysis of mask quality, such as overlap between masked regions and actual text regions?\n4.Why use cross-attention for multi-word queries but Hungarian matching for single words? Is there any reason for this ?\n5.The semantic query subset contains only 25 queries and 1000 images - is this scale sufficient for reliable evaluation, and how might this limited size affect the model's generalization ability on semantic queries?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1 The proposed progressive vision embedding method is well-motivated. Leveraging the zero-shot capability of VLMs to enhance scene text retrieval tasks is very meaningful, and I think this is a mainstream trend \n2 The highlight of this paper is that it addresses practical problems: high annotation cost of bounding boxes and difficulty in unified multi-query processing.\n3.Well-structured paper with detailed method descriptions and helpful figures/tables for understanding\n\nWeaknesses:\n1.Still exhibits clear limitations in handling extremely small and dense text, which somewhat restricts the method's applicability\n2.Despite the claimed advantages, Tables 4 and 5 reveal that MSTAR still underperforms compared to some box-based methods on specific metrics and datasets. For instance, in Table 4, MSTAR shows lower performance than TDSL on certain datasets like CTR, and in Table 5, it lags behind fine-tuned box-based methods (marked with *) such as TG-Bridge and Deepsolo on several benchmarks.\n3.The MIM module design has high complexity but limited performance improvement.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful feedback on our work! \n\n---\n\n***W1: Still exhibits clear limitations in handling extremely small and dense text, which somewhat restricts the method's applicability***\n\nWe appreciate the reviewer's comment. While extremely small and dense text remains inherently challenging for current models, our work takes a concrete step forward within the box-free paradigm. For instance, on the CTR dataset, which features small and complex text layouts, our model achieves a MAP of 60.13%, clearly outperforming prior methods such as SPTSv2 (48.39%) and BLIP-2 (45.75%). \n\nWe acknowledge that extremely small text remains an open problem for box-free method, our method brings unique advantages: it not only eliminates the box annotation cost, but also broadens the multi-query scene text retrieval applications. \n\nTherefore, we see this as a meaningful advancement in addressing fine-grained visual text understanding within box-free methods.\n\n---\n\n***W2: Despite the claimed advantages, Tables 4 and 5 reveal that MSTAR still underperforms compared to some box-based methods on specific metrics and datasets. For instance, in Table 4, MSTAR shows lower performance than TDSL on certain datasets like CTR, and in Table 5, it lags behind fine-tuned box-based methods (marked with \\*) such as TG-Bridge and Deepsolo on several benchmarks.***\n\nFor the CTR dataset that contains many extreme dense and small text instances, box-based method still hold advantages, as discussed in W1.\n\nWe follow the common practice used in text spotting methods such as TG-Bridge and Deepsolo. For these methods indicated as \\*, each dataset is evaluated using weights that are specifically fine-tuned on its corresponding training set. In contrast, our MSTAR model employs a single weight to evaluate all the datasets. Given that text spotting models benefit from dataset-specific fine-tuning while ours does not, a slight performance gap on these datasets is reasonable. Nevertheless, our method achieves comparable average performance against TG-Bridge (82.56 vs 82.94 ) across six datasets while offering over twice faster inference speed (14.2 FPS vs 6.7 FPS).\n\n---\n\n***W3: The MIM module design has high complexity but limited performance improvement.***\n\nMIM enables explicit one-to-one alignment without requiring large-scale contrastive training, making it a general and extensible component for multi-query retrieval tasks. MIM module is lightweight and introduces minimal computational overhead. MIM brings an average 1.07% improvement across all 7 benchmarks, and notably +2.19% on STR and +1.35% on CTW. \n\n---\n\n***Q1: In Fig. 1 b, while it's understandable that VLMs tend to overlook detailed text instances, why do they still maintain high attention scores for regions after covering the salient text areas? I.e., after masking \"Dead lakes park,\" the attention score remains as high as 96%.***\n\nThis is because the masking operation is designed with transparency, rather than completely removing the content. As shown in Fig. 1(b), although we mask out the salient text region (\"Dead Lakes Park\"), they remain visible. This process is similar to the design of SAS module which weakens the influence of salient regions to encourage the model to attend to overlooked fine-grained text instances. We will clarify these details in revised versions.\n\n---\n\n***Q2: In Tab. 8, why choose T=1 as the final configuration? Table 8 shows better performance at T=3, but authors chose T=1 for efficiency reasons - is this trade-off reasonable?***\n\nThe choice of T balances effectiveness and efficiency, as shown in Table 8. Increasing T from 0 to 1 yields a significant gain (e.g., +4.37% on CTR), while further increases up to T = 3 bring smaller improvements with noticeable drops in speed. We adopt T = 1 as the final setting for its favorable trade-off between performance and inference speed.\n\n---\n\n***Q3: Can you provide more quantitative analysis of mask quality, such as overlap between masked regions and actual text regions?***\n\nThank you for the suggestions! We evaluate the quality of our binary masks on the CTW dataset. During evaluation, we record the final-round binary mask used in the SAS module (T = 1). For comparison, we also obtain binary masks from the cross-attention weights of BLIP-2 using the same processing pipeline as ours.\n\nWe first extract the polygon coordinates of all text regions in each image to construct a binary ground-truth (GT) mask. We then compute the Intersection over Union (IoU) between each predicted mask and its corresponding GT mask. We calculate the mean IoU across 500 images from the CTW dataset. Additionally, we count the number of masks with IoU ≥ 0.5 as high-quality masks. The experimental results are shown below.\n\n| Method | Mean IoU | High-Quality Masks (IoU ≥ 0.5) |\n|--------|----------|-------------------------------|\n| BLIP-2 | 21.78    | 129 (25.8%)                   |\n| MSTAR  | 50.82    | 304 (60.8%)                   |\n\n\nThe results show that our approach generates significantly accurate text region masks compared to BLIP-2. The number of high-quality masks reaches 304 images out of all the 500 images.\n\n---\n\n***Q4: Why use cross-attention for multi-word queries but Hungarian matching for single words? Is there any reason for this ?*** \n\nWe consider each single word as an individual semantic instance and each learnable query is used to extract visual features corresponding to a single instance. This is like the DETR paradigm in object detection. Therefore Hungarian matching is sufficient and appropriate to establish one-to-one correspondence. However, the multi-word queries contain multiple semantic units. To handle this, a lightweight cross-attention layer is used to aggregate all the vision embeddings.\n\n---\n\n***Q5: The semantic query subset contains only 25 queries and 1000 images - is this scale sufficient for reliable evaluation, and how might this limited size affect the model's generalization ability on semantic queries?***\n\nThanks for the comment! First, compared to previous datasets, our dataset contains a comparable number of queries and images. For example, when compared with PSTR [1], our dataset has a similar number of images (1080 vs 1000). Compared with CSVTR [2], the number of images is also close (1.6 k), and the number of queries is similar (23 vs. 25). During the collection process, we predefined a variety of query types. Data collectors were instructed to gather images from scenes such as product displays, documents, restaurants, and street views. Finally, a manual filtering step was conducted to ensure the correctness and quality of the collected data.\n\n[1] Zeng, Gangyan, et al. \"Focus, distinguish, and prompt: Unleashing clip for efficient and flexible scene text retrieval.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n[2] Wang, Hao, et al. \"Scene text retrieval via joint text detection and similarity learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021."}]

Author rebuttal:

Meta-review: This paper presents a box-free scene text retrieval method that eliminates the need for bounding box annotations by employing progressive vision embedding and style-aware queries to enhance text representation and vision-language alignment. It also introduces a new dataset to serve as a benchmark for multi-query retrieval. The authors report convincing performance gains across seven public datasets for both scene text retrieval and scene text spotting tasks. The authors provided a strong rebuttal that effectively addressed most of the concerns raised by the reviewers. The paper received three "Borderline Accept" scores and one "Borderline Reject." The reviewer who assigned the "Borderline Reject" expressed the view that scene text retrieval is a very specific and narrow task within the broader retrieval community, which may limit its interest for a general audience. After carefully reviewing all comments, the rebuttal, and the discussion, the AC recommends acceptance.
Decision: Accept (poster)