Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Authors: Gengluo Li
Huawen Shen
Yu Zhou
Venue: ICML 2025 poster (2025)
Keywords: Scene text retrieval
Multimodal retrieval
Text understanding
Text layout
Primary area: applications->computer_vision
TL;DR: We introduce a new benchmark for Chinese scene text retrieval, highlighting the limitations of previous methods and proposing an approach that outperforms existing techniques.

Abstract: Chinese scene text retrieval is a practical task that aims to search for images containing visual instances of a Chinese query text. This task is extremely challenging because Chinese text often features complex and diverse layouts in real-world scenes. Current efforts tend to inherit the solution for English scene text retrieval, failing to achieve satisfactory performance. In this paper, we establish a Diversified Layout benchmark for Chinese Street View Text Retrieval (DL-CSVTR), which is specifically designed to evaluate retrieval performance across various text layouts, including vertical, cross-line, and partial alignments. To address the limitations in existing methods, we propose Chinese Scene Text Retrieval CLIP (CSTR-CLIP), a novel model that integrates global visual information with multi-granularity alignment training. CSTR-CLIP applies a two-stage training process to overcome previous limitations, such as the exclusion of visual features outside the text region and reliance on single-granularity alignment, thereby enabling the model to effectively handle diverse text layouts. Experiments on existing benchmark show that CSTR-CLIP outperforms the previous state-of-the-art model by 18.82% accuracy and also provides faster inference speed. Further analysis on DL-CSVTR confirms the superior performance of CSTR-CLIP in handling various text layouts. The dataset and code will be publicly available to facilitate research in Chinese scene text retrieval.

Beyond Cropped Regions:
New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Gengluo Li
Huawen Shen 12 Yu Zhou'
Abstract
Qaey +KEEAA Harbbebli
Chinese scene text retrieval is a practical task that aims to search for images containing visual in- stances of Chinese query text. This task is ex- often tremely challenging because Chinese features complex and diverse layouts in real-world scenes. Current efforts tend to inherit the solution for English scene text retrieval, failing to achieve satisfactory performance. In this paper, we estab- lish a Diversified Layout benchmark for Chinese Street View Text Retrieval (DI-CSVTR), which is specifically designed evaluate retrieval per- formance across various text layouts, including vertical, cross-line, and partial alignments. To ad- dress the limitations in existing methods, we pose Chinese Scene Text Retrieval CLIP (CSTR- CLIP), novel model that integrates global vi- sual information with multi-granularity alignment training. CSTR-CLI applies two-stage training process overcome previous limitations, such the exclusion of visual features outside the text re- gion and reliance on single-granularity alignment, thereby enabling the model to effectively han- dle diverse text layouts. Experiments on existing benchmark show that CSTR-CLIP outperforms the previous state-of-ther -fttt--arttt model by 18.82% ac- curacy and also provides faster inference speed. Further analysis on DL-CSVTR confirms the su- perior performance of CSTR-CLIP in handling various text layouts. The dataset and code will be publicly available to facilitate research in Chinese scene text retrieval.
Text
Text
Votkal
BANK BANK
Cross Lasr
BANK BANK
Partal
Visual Embeedin
Embedding
Embeeding
Teatt Eambeling
Text Eambding
Text Eembding
"BANK"
*HIT (c) Clas
Englishsmme natiorl
(3d Puxtous Clinae soone toxtintiooal
Figure Pipeline comparison between (a) English scene text re- trieval, (b) Chinese scene text retrieval adopted in previous work, and (c) Our proposed Chinese scene text retrieval framework is based on full image information and multi-gran multi-gramuulaitty alignment. The patterns in circles represent text layout forms.
1. Introduction
Text is important object in scene images, and scene text related research topics including detection (Wang et al., 2022; Cao et al., 2025), recognition (Yang et al., 2025; Zhang et al., 2025), spotting (Lyyu et al., 2025a; Wei et al., 2022), understanding (Shen et al., 2025; Zeng et al., 2023) and processing (Zeng et al., 2024b; Shu et al., 2025) have drawn increasing attention in recent years. Among them, scene text retrieval is important topic of information retrieval, which involves searching for images that contain visual instances of given query text within a collection of natural images (Mishra et al., 2013; Mafla et al., 2020). As texts in images generally convey valuable information, this task has been widely used in many applications, such as multimedia content retrieval, product recommendation. and automatic navigation (Karaoglu al., 2016; Bai et al., 2018 8; Song et al., 2019).
'Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China "School of Cyber Security. University of Chinese Academy of Sciences, Beijing, China 'VCIP TMCC & DISSee, College Computer Science, Nankai University, Tianjin, China. Correspondence to: Yu Zhou <yzhou@ mankai.cdu.cc2..
In recent years, scene text retrieval has witnessed tremen- dous progress, while most of the existing methods focus on the language patterns of English. Whether these methods can be transferred to other language scenarios (especially
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
non-latin scripts) has not been fully explored. More specifi- cally, as one of the most widely used non-latin languages, Chinese scene text retrieval differs distinctly from English scene text retrieval. As shown in Figure ((a), in English setting, the query term is typically a single word, and text in images exhibits clear separations. As such, English scene text retrieval is essentially a simple local matching problem, which can be solved by measuring the similarity between the detected region and the query word. In contrast, in Chinese setting, there are no separations among words of the same sentences, the query terms can be any combina- tion of consecutive characters, and Chinese has more highly variable layouts in real scenes. As shown in Figure 1(b), detection results often do not completely match the query words. Therefore, it is difficult to adapt the scene text re- trieval pipeline from English Chinese straightforwardly.
span rows or columns, and (iii) partial layouts where query terms are connected to other text. DL-CSVTR combines diverse layouts to evaluate the ability of models to handle various text instances, simulating the requirements in real Chinese scene text retrieval applications.
Following it, we are committed to enhancing model perfor- mance on diverse text layouts by freeing the model from the limitations imposed by cropped regions on its visual perception range, which we believe is key to addressing the challenges of these layouts. To this end, we take ad- vantages of the Contrastive Baangugee-mmgg Pre-training (CLIP) model to retain full-image information, and convert the problem from text regions matching to global semantics enhanced layout patterns learning, as shown in Figure (cc). Specifically, our proposed Chinese Scene Text Retrieval CLIP model adopts two-stage training paradigm. The goal of the first stage is to teach CL.IP to focus on specific areas and enhance its OCR capabilities. During it, triplet inputs including the original image, the text segmentation map and the corresponding text are constructed, which are employed to train CLIP with a graaulartyy alignment method. In the second stage, the model goes further into multi-granularity alignment, so to learn the ability of per- ceiving diverse layouts. We apply the random granularity alignment algorithm to process the segmentation map and text in the triplets. producing many layout patterns for the model to learn. Mcanwhile, global image features derived from the original CLIP model are injected in multi-scale lay- ers of CSTR- CLIP, enabling the model to perform retrieval under the guidance of context.
Targeting at the Chinese scene text retrieval task, Wang et al. (Wang et al., 2021) establish the CSVTR benchmark, and several studies have demonstrated promising performance on this benchmark. However, it can not well reflect the re- trieval capabilities of models in real world. The query terms in this benchmark predominantly appear independently and are primarily horizontally oriented, neglecting the character- istics of diverse text layouts in Chinese scene text retrieval. In addition, most existing Chinese scene text retrieval mod- els adopt paradigms developed for English scenes, utilizing cropped text regions from text detection results to represent the visual information of images, which brings several limi- tations. On the one hand, this method inevitably results in the loss of context, that is, the global semantic information can not contribute to the retrieval process, and parts of the query term may be lost. On the other hand, since redundant elements may be included in the detected boxes, the model is easily disturbed by irrelevant content. Moreover, previous methods strictly align the features of a text region with all the text it contains during training. This alignment manner will undoubtedly cause some ambigu- ity, impairing models' ability to perceive and distinguish text. In Chinese scene text, the diverse combinations and layout forms of characters present significant challenges to previous solutions based on cropped regions. Therefore, en- hancing the ability of scene text retrieval models to handle varied layouts is crucial for advancing scene text retrieval.
Experiments on the previous Chinese scene text retrieval benchmark CSVTR show that CSTR-CLIP improves re- trieval accuracy by 18.82% over the best previous method, while also delivering faster inference speed. Further- more, both quantitative and qualitative experiments on DL- CSVTR confirm the effectiveness and generalization ability of CSTR-CLIP in handling diverse text layouts.
In summary, the contributions of this paper include:
We conduct a thorough analysis of previous benchmark datasets and methods for Chinese scene text retrieval, identifying a significant gap in their ability handle diverse text layouts. To address it, we propose new benchmark, DL-CSVTR, which includes Chinese text image data for various layouts, to evaluate models' retrieval capabilities in real scenarios.
Since previous studies have not addressed the specific chal- lenges of Chinese text retrieval, we propose new bench- mark and a scene text retrieval model that extends beyond cropped regions. As first contribution, we introduce DL- CSVTR, a new Diversified Layout benchmark for Chinese Street View Text Retrieval. We collect scene images con- taining visual instances of query terms in three of the most common text layouts, in addition to the horizontal layout, including (i) vertical layouts where the query terms are ar- ranged vertically, (ii) line layouts where query terms
We introduce ESTR-CLIP. a novel paradigm for Chi- nese scene text retrieval. CSTR-CLIPP moves beyond the previous approach of text regions cropping, ex- pands the model's visual receptive field by retaining full-image information, and enhances perception flexi- bility through multi- maaniliraaularity alignment.
2

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
2015; Wilkinson & Brun, 2016; Gómez et al., 2018; Mafla et al., 2021), while they still suffer from the suboptimality of manually designed embeddings (Zhou et al., 2022b). Sub- sequently, cross-modal embeddings are proposed to unify visual and textual representations within a common feature space, achieving strong retrieval performance and speed (Gómez et al., 2017; Mhiri et al., 2019; Wang et al., 2021; Zeng et al.. 2024a). Recently, visual embedding methods propose to transform query words into visual representations for retrieval via visual matching (Wen et al., 2023; Luo et al., 2024), achieving remarkable performance but at the cost of slower computational speeds. All these approaches rely on text detection to crop text regions for feature extraction. A Although cropping enhances the visual features of the text region, the loss of semantic information outside this region has limited further improvements in scene text retrieval.
Harizontal: 92.62%
Vertival: 3.53%
CSVTR
Cross-Line: 1.07%
Partial: 2.75%
77
Text bbox containing the query
Figure 2. I Layout distribution of the visual instance of the query word in the image in CSVTR.
Our experiments demonstrate that the proposed method not only achieves state-of-the-ar performance on the existing CSVTR dataset but also surpasses previous models in retrieval capabilities on the newly introduced DL-CSVTR benchmark.
CLIP's Attention Guidance and OCR Capability. CLIP (Radford et al., 2021) is vision-language model trained on large-scale data, possessing powerful representation capabil- ities (Materzyúska et al., 2022: Yu et al., 2023). Moreover, researchers find that many high-similarity image-text pairs in the LAION-2B dataset (Schuhmann et al., 2022) include visual instances of captions within the images (Lin et al., 2024) and synthetic text images can be utilized as visual prompts to enhance image classification performance (Li et al., 2022: Shi & Yang, 2023). These studies suggest that CLIP possesses potential OCR functionality and is well- suited for retrieval tasks (Luo et al., 2022; Baldrati et al., 2023; Saito et al., 2023). However, CLIP's perception of- ten involves all image elements, resulting in a bias towards prominent objects (Xing et al., 2023). To enhance region awareness, methods like ReCLIP (Subramanian et al., 2022) crop images using bounding boxes, though this can lead to the loss of visual information. Red-Circle (Shtedritski et al., 2023) adds contours to guide CLIP's attention, while MaskCLIP (Zhou et al., 2022a) and AlphaClip (Sun et al., 2024) use masks to focus on specific regions, guiding CLIP to focus target areas while preserving global context. Inspired by these works, our approach leverages CLIP to extract full-image information and facilitate scene text re- trieval with CLIP's intrinsic text knowledge. Additionally, text location information is exploited to guide the model's perception area.
2. Related Work
Scene Text Retrieval Benchmark. Mishra al. (Mishra et al., 2013) is the first work to introduce the scene text retrieval task with the IIIT Scene Text Retrieval benchmark, which comprises numerous scene images and predefined query terms linked to specific images. The model accuracy is evaluated using Mean Average Precision (mAP), and speed is measured with Frames Per Second (FPS). Later researchers enhance this by using the Street View Text and TotalText datasets, creating richer benchmarks with text annotations as query terms (Wang & Belongie, 2010; Ch' ng & Chan, 2017). To examine the retrieval problem in other language setting, Wang et al. (Wang et al., 2021) introduce the CSVTR dataset for Chinese scene text retrieval, where query terms typically appear independently and are arranged horizontally, as shown in Figure 8. It ignores the challenges that the diverse text layout of Chinese brings to the scene text retrieval task.
Scene Text Retrieval. For scene text retrieval, a straight- forward solution is to use text detection (Shu et al., 2023) and recognition( (Qiao et al., 2020b; 2021) or end-to-end spotting (Lyu et al., 2025b) models to extract characters or words from images, simplifying the problem traditional text retrieval (Mishra et al., 2013; Jaderberg et al., 2016; Liu et al., 2020; Qiao et al., 2020a; Liao et al., 2020). How- ever, this approach often leads to error accumulation and limited performance (Huang et al., 2024). Some studies design manual text embeddings to convert characters into vector representations for improved retrieval robustness (Al- mazán et al., 2014; 4; Ghosh et al., 2015; Ghosh & Valveny,
3. DL-CSVTR Benchmark
Based on our findings, we propose Diversified Layout benchmark for Chinese Street View Text Retrieval (DL- CSVTR), which considers the diverse text layouts of query terms' visual instances in Chinese scene text retrieval. Specifically, it includes data from three common scene text layout types, where the visual representation of query terms strictly adheres to their respective layouts: vertical, cross-
3

---

We observe that different Chinese text layouts lead to vary- ing granularities in the outputs of text detectors. For exam- ple, a cross-line layout may produce detection boxes that capture only portions of the query terms, while a partial lay- out might include additional characters. Besides, vertically arranged text is often associated with surrounding visual elements, such as nearby buildings or scenery. Previous scene text retrieval models typically crop text regions based on detector results, discarding all visual information out- side the detection boxes and thereby losing valuable context. Additionally, these models employ single-graullartty align- ment, strictly aligning the visual features of a text region to the textual features of all text contained within, thereby lim- iting the model's 's broader perceptual scope. Consequently, our proposed DL-CCVTR benchmark introduces new chal- lenges to assess the model's ability of handling diverse text layouts, with the data being used exclusively for testing purposes. Detailed settings of the query terms and image Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
(c) CSVTR Horianta
(b) DL-CSVTRVV (c) DI-CSVTR-CL (d) DI-CSVTR-P
collect 1,250 images with 25 query terms where the text detector's detected regions might include unrelated extra not unrelated characters.
o
Croa-in
Dala Sourze & Corresponding Text Layoul
Example
Visual insta of query "MRK""
Text quary
370/
130/33
12/0/
Number of images/ guortes
Figure 3. Common Chinese text layouts, where the visual repre- sentation of query terms is based on the text detection model's cropped results. In category (d), the red-highlighted area indicates the corresponding query term.
line, and partial, shown in Figure 3.
To ensure the quality of annotations and minimize the im- pact of individual annotator bias, we adopt the strategy of simultaneous data collection by three annotators. First, we define 89 query terms for the three layouts. These query terms are primarily common conceptual nouns, including but not limited trademark names, building names, and idioms. The details of the specific query word settings can be found in the supplementary material. Subsequently, the three annotators use these predefined query terms to search for images containing the specified text layouts of the query terms' visual instances. The results from all three annota- tors are then combined. After combining the results, the annotators screen the image set to remove duplicates and images where the query terms' visual representation did not match the specified text layouts, ensuring the uniqueness of the text layout for each query term in the images. The specific descriptions of each text layout are as follows:
visualizations can be found in the supplementart material.
4. Method
This section introduces our proposed scene text retrieval model, Chinese Scene Text Retrieval CLIP (CSTR-CLI)), which leverages goati-graanllarity perception guided by text regions and full-image information understanding capabili- ties. Additionally, it covers the training data-based random granularity alignment algorithm and explains how the model is applied to downstream tasks in scene text retrieval.
To overcome the limitations of the text region cropping paradigm, we use the entire image for feature extraction, expanding the visual receptive field. To distinguish different text labels in the image, we synthesize segmentation maps by annotating the position information of the text labels, using them to guide CLIP's focus specific regions. A two-stage training process is designed to gradually equip CLIP with the ability to handle scene text retrieval across diverse text layouts. The model structure is illustrated in Figure 4.
Vertical. Vertical text is common in real-world scenes, characterized by text arranged from top to bottom, as shown in Figure 3(b). We collect 370 street view images containing vertically arranged text with 31 unique query terms. All these queries are visually represented in vertical layout within the images.
Stage 1: Training CLIP's OCR and regional perception capabilities. As shown in Figure 4(a), the model is initial- ized with pre-trained Chinese CLIP weights (Yang et al., 2023), and Text Position Convolution is introduced to en- hance the regional perception capabilities. During training, we keep the text encoder Ey frozen.
Cross-Line. Cross-line text layouts occur when a concep- tual noun spans multiple rows or columns, as shown in Figure 3(c). We collect 450 street view images featuring cross-line text layouts with 33 unique query terms. These queries are visually represented in either cross-row or cross- column layouts within the scene images.
Given an input image E Hxx3 it first embedded with the CLIP RGB Convolutional layer CONURGB- Then, we generate segmentation map G E Hxxxx1 for each text label in L, highlighting the pixels in the text area while
Partial. Unlike English words, Chinese characters typically do not have distinct separations. When query terms appear in a long sentence, text line detection often includes charac- ters beyond the query terms, as shown in Figure 3(d). We
4

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Randon Mask
EGB
Image Encoder Ei
Text Encoder Er
LNCE
"J
Random Splicing
Kandons Expond
Twrthee
7
Foor
UrT
Fix
Cg
"JIDRIB"
(d) Stage I: Focuses un Training CLIP's OCR and Regional A Altentinn Capabilities
(c) Random A Ligument G Granukrity Procxsing (RAGP)
=
Image Enunder B
Eawoad Lowel
Townder Sapper xx
Place
FlL
Place
Pl
v
LNCE
d/Toscion
Text Encoder
Pasten Lager
Faskon Layer
Podien Lager K
Tiesten
Enncade Lawer La
Pings
RAGP
Frondee
Danoker
Image Encuder Eatgh
Stage 2: Training with Global Information Integration and Multi-Gramllrity Aatrannllartyllle Alignment.
Figure Two-stage training framework the CSTR-CLIP model, including: (a) Training model's region attention and OCR capabilities based on single-granularity alignment; (b) Adapting CSTR-CLIP to diverse text layouts using random multi-granularity alignment and integrating global features. (c) The Random Alignment Granularity Processing (RAGP) module stage 2.
setting the background pixels to zero. We design a single- channel convolutional layer, ConVTentpos- that accepts G as input to obtain the guided embedding. After that, these two embeddings are fused:
This stage enhances CLIP's OCR capabilities and its per- ception guided by the specified regions. The model ini- tially trained with synthetic data and then fine-tuned using real-world scene data. It can direct perception speci- fied regions through ConvTezztoss However, the single- granularity alignment method used during training leads to losing some visual features in non-guided areas.
Fogg = COnURGB(I) + Conveeeztpod
Stage 2: Training with Global Information Integration and
Next, Flg is fed into the CLIP image encoder E, to obtain the image feature 1/z E IRd, where d is the dimension of the CLIP embedding space. The corresponding text is fed into the text encoder Er to obtain the text feature u/ E Rdd:
Alignment. We utilize the frozen original Chinese CLIP visual encoder Earigin to enhance our first stage model, as illustrated in Figure 4(b). The scene image I is also fed into Eurigin simultaneously. For each layer 1, the intermediate features Ft from E, are augmented by the corresponding global features F from Eorigin
vr, NT (F
During training, for each image I, we generate triplets {Li, Gik, } by iterating over its text box annotations, where the image index and is the text annotation index. The number of text line annotations determines the number of triplets generated. We use all triplets generated from the images the training set, apply single-granularity align- ment method to align text regions with their contents, and train the model using NCE Loss:
Pluse
F! Flous + FusionLLayer (Forigin)
where the Fusion Layer is 1xl convolutional layer.
This process aims to recover the visual feature loss in non- perception regions from the first stage, thereby enhancing the model's understanding of global information. This stage exclusively uses real-world scene data for training.
Random Alignment Granularity Processing (RAGP). To augment the training data for multi multi-granularity alignment, we apply RAGP on segmentation maps and text inputs, as lustrated in Figure 4(c). According to our previous analysis, single-gran -ranularraul alignment limits the model's perception
explett ////T)
LNCE =
log E exp(1t) /
(3)
N
where T the temperature parameter.
5

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Gallery Set
combination of synthetic and real data for model training.
Gu
CSTR-CLIP
Image Encoder
Synth Text-CH. Following SynthText-900K (Gupta et al., 2016), we creat the SynthText-CHE dataset with 300K samples, recording bounding box coordinates to generate segmentation maps for the first training stage.
Text Detector
CSTR-CLIP Text Encoder
query
ReCTS. The ReCTS (Zhang et al., 2019) dataset com- prises annotated Chinese text images in natural scenes, Segmentation maps are generated from these annota- tions for both training stages. serves as the data for the second training stage.
Figure 5. The retrieval pipeline of CSTR- -LIP.
the entire text within the guided region, thereby reducing its flexibility. In cross-line and partial layouts, the correspond- ing text region may miss some queries or include redundant text, making single-graulant
Test Datasets. The test data comprises the existing Chinese scene text retrieval benchmark CSVTR and our proposed DL-CSVTR.
alignment less than ideal. The key to solving this problem is to enable the model to flexibly perceive text elements inside and near the text re- gion, disrupting the exact match between the segmentation map G.s and the corresponding text Tis during training.
CSVTR. This dataset includes 1,667 images which are gathered using Google Image Search, corresponding to 23 predefined queries. The visual appearance of the query terms in the images is primarily in horizontal layout and presented independently.
We truncate Tik by masking m characters from both ends, ensuring a minimum length of 2, with the deletion proba- bility controlled by a hyperparameter B. We assume that semantically related text elements are spatially close in the image. For Tik we identify the text Tip whose bounding box centroid is closest to Ta and extend it in the direction connecting their centroids, with the probability of adding text controlled by hyperparameter a. For the segmenta- tion map G.k. we locate the segmentation map Gp whose bounding box centroid is closest to Ta and concatenate them, with the probability of concatenation controlled by hyperparameter 0. In general, the text first randomly expanded and then randomly masked, and the processing of the segmentation map is independent of the text processing,
The DL-CSVTR dataset proposed in this paper includes three types of text layout bench- marks, with a total of 2,070 images and 89 predefined query terms. This dataset will be used to evaluate the model's retrieval ability across different text config- urations. The tests will be conducted on the subsets corresponding to the three types of text layouts.
DL-CSVTR.
Baselines. We use previously published scene text retrieval models tested on CSVTR baselines. We reproduce and evaluate the most competitive methods on DL-CSVTR. Since earlier methods do not utilize the CLIP pre-trained model, we replace the backbones in previous works (Wang et al., 2021; Luo et al., 2024) with the pre-trained CLIP model to ensure a fair comparison. These methods repre- sent the state-of-the-art in cross-modal and visual embed- ding paradigms. The source of training data the same as CSTR-CLIP, with the text area cropped according the annotations for the training set.
CSTR-CLIP for Inference. As illustrated in Figure 5, we use PaddleOCR to detect text, and RAGP is not applied in the inference stage. Detected text regions are converted into segmentation maps and paired with the original image, forming the gallery set. If no text regions are detected, a fully highlighted image is used as the segmentation map. All related pairs in the gallery set are processed by the vi- sual encoder to extract visual features, and the query word is encoded into text features by the text encoder. For an image, multiple binary groups are formed with the various text regions identified by the detector, resulting in multiple visual features guided by different text regions. The highest similarity score between these visual and text features deter- mines the final score for the image's visual representation.
Implementation Details. and the small text in scene images, we expand the input resolution of CSTR-CLIP to 640x640. Yet, this will lead to inappropriate positional encoding in the self-attention layer that processes the features extracted by visual extraction module, replace them with learnable ones, initialized using nearest neighbor interpolation. The visual extraction module utilizes the pre-trained ResNet50 version.
Due to CLIP's limited input size
5. Experiments
We optimize CSTR-CLIP using the Adam optimizer with an initial learning rate of 1e-6 and a batch size of 48. The first stage is trained for 8 epochs (6 with synthetic data and
Training Datasets. We follow the training setting of previ- ous studies (Wen et al., 2023; Luo al., 2024), employing
6

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
2 with real data), while the second stage is trained for 10 epochs. In RAGP, we the hyperparameters controlling the randomness to 3 = 0.2, = 0.3, and 0 = 0.2. Models are trained on an NVIDIA A6000 GPU and tested on a GTX 1080 GPU.
ing. For vertical layouts, our method outperforms cropped region-based methods by leveraging full-image informa- tion, demonstrating that understanding information beyond cropped regions is crucial for improving the model's under- standing of vertical text layouts. In cross-line and partial layouts, the model, limited by single-granularity alignment, does not surpass the CLIP baseline after the first stage. How- ever, in the second stage, the fusion of full-image informa- tion with multi-granulan text regions, overcomes the limitations of single-granularity alignment and non-guided region information loss from the first stage, resulting in impressive performance.
5.1. Performance on CSVTR benchmark
perception training, guided by
Method
mAP %
FPS
Mishra et alMMishra et al., 2013)
4.79
0.10
TDSL (Wang et al., 2021)
60.23
12
VSTR (Wen et al., 2023)
63.17
11
Luo et al(Luo et al., 2024)
69.75
11.2
CLIP (a40x640)Raddford ct al., 2021)
57.32
47.6
5.3. Ablation Study
TDSL (Wang et al., 2021)*
78.36
12.3
Luo et al(Luo et al., 2024)*
80.74
10.5
In this section, we conduct a series of ablation experiments based on our model training stages. The detailed results are presented in Table 3.
CSTR-CLIP
88.57
21.5
Table 1. Comparison with existing methods on CSVTR. 1 repre-
Effectiveness of Stage settings. We conduct an abla- tion study on the Stage 1 settings to evaluate their effec- tiveness. (See #1, #2 and #3) When the visual encoder is frozen, ConUTestpos can be considered a data augmentation method, adapting the image to the visual encoder's 's attention bias through performs well on CSVTR, where query terms typically ap- pear independently and a horizontal layout, resulting in better performance due accurate text detection. However, in the more diverse settings of DL-CSVTR, the performance declines due to suboptimal detector outputs, particularly in partial and cross-line layouts where only parts of the query terms or extraneous elements are captured.
sents the version that replaces the backbone with the CLIP encoder. We highlight the best and the second results.
We compare our model with previous models CSVTR, and the experimental results are shown in Table 1. The results clearly demonstrate that our model outperforms the previous methods, improving retrieval accuracy by 18.82% while maintaining competitive inference speed. Addition- ally, our method continues to excel even when the encoders of of-te-of-the-art models based on cross-modal and visual embeddings are replaced with the CLIP encoder as an ad- ditional baseline. We attribute this improvement to our method's 's ability to retain visual features outside the text regions from a global image perspective. In scenarios where the detector fails to detect the text regions or the text regions in the image have poor clarity, our full-image based method leverages rich visual context information to achieve superior retrieval capabilities.
alignment. This approach
Involving the visual encoder Ey in model training enhances performance across all datasets (See #2 and #3). This im- provement results from fine-tuning the OCR capabilities and enhancing ConUTeatpess" 's guidance function the vi- sual encoder level. The gains are especially notable on CSVTR and DL-CSVTR-V, where jointly training the image encoder E, and ConUTeztpos results in optimal attention guidance. However, improvements on DL-CSVTR-CO CC and DL-CSVTR-P are less pronounced, as the single-granularity alignment method overly restricts the model's 's focus, result- ing in poorer performance when only part of the query's s visual representation or irrelevant elements are included in the guidance area.
5.2. Performance on DL-CSVTR benchmark
As shown in Table 2, CLIP demonstrates strong retrieval ability across all DL-CSVTR benchmarks, highlighting the importance of perceiving beyond cropped regions for un- derstanding diverse text layouts. While methods based on the cropped regions paradigm surpass the CLIP's retrieval ability in vertical layouts, they perform poorly in cross-line and partial layouts. This indicates that the cropped regions paradigm struggles address the challenges posed by these more complex configurations.
Effectiveness of Stage 2 settings. For the Stage 2, the ab- lation results of each setting are also reported (See #3, #4, #5 and #6). The inclusion of full-image features on CSVTR leads some improvements, but RAGP does not yield sig- nificant gains, query terms in CSVTR typically appear independently and in a horizontal layout. In DL-CSVTR-V V, the addition of full-image information brings significant im- provement, validating the importance of full-image layout information in understanding vertically aligned text. For
However, after single-granularity training in the first stage, our method surpasses previous methods based on the cropped regions paradigm across all DL-CSVTR bench- marks, and further surpasses all baselines to achieve the best retrieval performance after the second stage of train-
7

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Method
DI-CSVTR-V (mAP %)
DI.-CSVTR-CL (mAP %)
DI-CSVTR-P (mAP %)
Luo et al(Luo et al., 2024)
39.87
21.98
13.95
CLIP (640x640) (Radford et al., 2021)
55.48
54.43
37.31
TDSL(Wang et al., 2021)*
62.51
40.46
24.88
Luo et al(Luo al., 2024)
52.73
34.23
29.91
CSTR-CIL (Stagel)
74,50
45.98
33.25
CSTR-CLIP
84.44
65.56
61.85
Table 2. Comparison with existing methods on DL-CSVTR. We highlight the best and second results.
represents the version that replaces the backbone with the CLIP encoder.
Stage1
Stage2
mAP(%)
#
ConuTextpos
IE
Global features
RAGP
CSVTR
DI-CSVTR-V
DI-CSVTR-CL
DL-CSVTR-P
x
x
x
X
57.32
55.48
54.43
37.30
x
x
X
59.50
53.17
42.15
30.70
3
V
V
X
X
86.25
74.50
45.98
33.25
4
v'
X
88.41
83,91
57.82
43.19
5
86.31
74.01
60.08
52.51
V
88.57
84.44
65.56
61.85
Table 3. Ablation experimental results. In Stage 1, we perform ablation studies the use of Conerrarries and the inclusion of the image encoder (IE) training. Stage 2, we ablate the inclusion of global features from the original CL.IP model use multi-granularity alignment processing. We highlight the best and the second results.
DL-CSVTR-CL and DL-CSVTR-P, integrating full-image information and random granularity alignment processing significantly enhances the model's ability to retrieve text across rows and in partial layouts. RAGP refines the model's attention to partial text within the guided area and external text content, while full-in full-image information enhances the model's s overall perception and understanding of the im- age content. As shown in Figure 6, we visualize the fused features of the intermediate layers using the merged chan- nel visualization method. The visualization results clearly indicate that after the Stage 2 training, CSTR-CLA not only enhances its perception of the entire image but also improves its understanding of the text information near the guided area and further refines the internal text elements.
Query
Segmantation
Map
Top
Figure 7. An example of region-specified seene text retrieval. The green boxes in the search results indicate visual instances of the query terms.
Precs
Flased
Payez
Ruject
5.4. Interactive Region-Specified Scene Text Retrieval
(a)
Our approach paves the way for more diverse text retrieval methods, enabling region-specific retrieval. Specifically, Contreztpos allows the model to focus on user-defined regions, while multi-granularity alignment enhances flexibility of the model's perception, permitting users to custom-design segmentation maps. illustrated in Figure 7, by highlighting specific parts of the segmentation map, we direct the model's attention to the corresponding image region, with the visual examples of the query terms in the top-ranked results mostly appearing in the guided region. This enables our model to perform more accurate scene text retrieval with region-specifie guidance.
(b)
(e)
Figure 6. Visualization results of model intermediate layer fcatures, including original CLIP, (b) CSTR-CLIP after Stage training, (c) CSTR-CLID after Stage training.
8

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
6. Conclusion
dataset for scene text detection and recognition. In Pro- ceedings of the International Conference Document Analysis and Recognition (ICDAR), 2017.
This paper identifies the limitations the previous scene text retrieval paradigm in handling the diverse text layouts of Chinese. To tackle a new benchmark DL-CSVTR is proposed, which contains three types of common text layouts for evaluating models' retrieval capabilities in real scenarios. To overcome the limitations of the previous text region cropping paradigms in Chinese scene text retrieval, we propose the CSTR- ESTR-CLIP model. Our model is designed with a aaatr--rrnlllat-wwaarrr paradigm, leveraging the entire image while being guided by text regions. This ap- proach not only achieves the best retrieval accuracy and speed on previous benchmarks but also demonstrates signif- icant performance improvements under various text layout conditions on DL-CSVTR. The text region-guided design allows users to specify regions of interest for more detailed retrieval in practical applications.
Ghosh, S. K. and Valveny, E. Query by string word spotting based on character bi-gram indexing. In Proceedings of the International Conference on Document Analysis and Recognition (ICDAR), 2015.
Ghosh, S. K., Gomez, L.. Karatzas, D., and Valveny, E. Efficient indexing for query by string text retrieval. In Proceedings of the International Conference on Docu- ment Analysis and Recognition (ICDAR), 2015.
Gómez, I.., Rusinol, M., and Karatzas, D. Lsde: Leven- shtein space deep embedding for query-by-string word spotting. Proceedings of the International Conference Document Analysis and Recognition (ICDAR), 2017.
Gómez, L., Mafla, A., Rusinol, M., and Karatzas, Single shot scene retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Acknowledge
Supported by the National Natural Science Foundation of China (Grant NO 62376266 and 62406318), Key Labora- tory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University China, Beijing, China.
Gupta, A., Vedaldi, A., and Zisserman, A. Synthetic data for text localisation in natural images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Impact Statement
Huang, M., Li, H., Liu, Y., Bai, X., and Jin, L. Bridging the gap between end-to and-to-end and two-step text spotting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Jaderberg, M., Simonyan, K., Vedaldi, A., and Zisserman, A. Reading text in the wild with convolutional neural networks. International Journal of Computer Vision, 2016.
References
Almazán, J., Gordo, A., Fornès, A., and Valveny, E. Word spotting and recognition with embedded attributes. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2014.
Karaoglu, S., Tao, R., Gevers, T., and Smeulders, W. Words matter: Scene text for image classification and retrieval. IEEE Transactions Multimedia, 2016.
Bai, X.,
Yang, M., Lyu, P., Xu, Y., and Luo, J. Integrating scene text and visual appearance for fine-grained image classification. JEEE Access, 2018.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified rision-language understanding and generation. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
Baldrati, A., Bertini, M., Uricchio, T., and Del Bimbo, A. Composed image retrieval using contrastive learning and task-oriented clip-based features. ACM Transactions on Multimedia Computing, Communications and Applica- tions, 2023.
Liao, M., Pang, G., Huang, J., Hassner, T., and Bai, X. Mask textspotter v3: Segmentation proposal network for robust scene text spotting. In Proceedings the the European Conference on Computer Vision (ECCV), 2020.
Cao, T., Lyu, J., Zeng, W., Mu, W., and Zhou, Y. The devil is in fine-tuning and long-tailed problems: A new benchmark for scene text detection. In IJCAL, 2025.
Lin, Y., He, C., Wang, A. Wang, B., Li, W., and Shou, M. Z. Parrot captions teach clip to spot text, 2024. URL https: http///wxvvvvoorbb///2/.e
Ch'ng, and Chan, C. S. Total-text: comprehensive
14232.

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Liu, Y., Chen, H., Shen, C., He, T., Jin, L.,, and Wang, L. Abenet: Real-time scene text spotting with adaptive bezier-curve network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Qiao, Z., Zhou, Y., Wei, J., Wang, W., Zhang, Y., Jiang, N., Wang, H., and Wang. W. PIMNet: A parallel, iterative and mimicking network for scene text recognition. In ACM MM, pp. 2046-2004000 2066-005,5220 2021.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Luo, H., Ji, L.,, Zhong, M., Chen, Y. Lei, W., Duan, N., and Li, T. ClipAclip: An empirical study of clip for end-to- end video clip retrieval and captioning. Neurocomputing, 2022.
Saito, K., Sohn, K., Zhang, X., Li, C.-L., Lee, C.-Y., Saenko, K., and Pfister, T. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Luo, H., Ibrayim, M., Hamdulla, A., and Deng, Q. Visual and semantic guided scene text retrieval. The Journal of Supercomputing, 2024.
Lyu, J., Wang, W., Yang, D., Zhong, J., and Zhou, Y. Arbi- trary reading order scene text spotter with local semantics guidance. In AAAI, volume 39, pp. 5919-5927, 2025a.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large- scale dataset for training next generation image-text mod- els. Advances in Neural Information Processing Systems, 2022.
Lyu, J., Wei, J., Zeng. G., Li, Z., Xie, E., Wang. W., Ma, C., and Zhou, Y. VextBlockV2: Towards precise-detection- free scene text spotting with pre-trained language model. TOMM, 2025b.
Shen, H., Li, G., Zhong. J., and Zhou, Y. LDP: Generalizing multilingual visual information extraction by language decoupled pretraining. In AAAI, volume 39, pp. 6805- 6813, 2025.
Mafla, A., Dey, S., Biten, A. F., Gomez, L., and Karatzas, D. Fine-grained image classification and retrieval by combining visual and locally pooled textual features. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision (WACV), 2020.
Shi, C. and Yang, S. Logoprompt: Synthetic text images can be good visual prompts for vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Mafla, A., Tito, R., Dey, S., Gómez, I.., Rusinol, M., Val- veny, E., and Karatzas, D. Real-time lexicon-free scene text retrieval. Pattern Recognition, 2021.
Shtedritski, A., Rupprecht, C.,, and Vedaldi, A. What does clip know about red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Materzyúska. J., Torralba, A., and Bau, D. Disentangling visual and written concepts in clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Shu, Y., Wang, W., Zhou, Y., Liu, S.. Zhang, A., Yang, D., and Wang, W. Perceiving ambiguity and semantics without recognition: An efficient and effective ambiguous scene text detector. In ACM MM, pp. 1851-1866 2023.
Mhiri, M., Desrosiers, C., and Cheriet, M. Word spotting and recognition via a joint deep embedding of image and text. Pattern Recognition, 2019.
Shu, Y., Zeng, W., Zhao, F., Chen, Z., Li, Z., Yang. X., Zhou, Y., Rota, P., Bai, X., Jin, L., et al. Visual text processing: A comprehensive review and unified evaluation. arXiv preprint arXiv:2504.2662. 2025.
Mishra, A., Alahari, K., and Jawahar, C. Image retrieval us- ing textual cues. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2013.
Qiao, L.. Tang, S., Cheng, Z., Xu, Y., Niu, Y., Pu. S., and Wu, F. Text perceptron: Towards end-to-end arbitrary-shaped text spotting, In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020a.
Song, H., Wang, H., Huang, S., Xu, P., Huang, S., and Ju, Q. Text siamese network for video textual keyframe detection. In Proceedings of the International Conference on Document Analysis and Recognition (ICDAR), 2019.
Qiao, Z., Zhou, Y., Yang, D., Zhou, Y., and Wang, W. SEED: Semantics enhanced encoder framework for scene text recognition. In CVPR, pp. 13528-13537, 2020b.
Subramanian, S., Merrill, W., Darrell, T., Gardner, M., Singh, S., and Rohrbach, A. Reclip: A strong zero-shot bascline for referring expression comprehension, 2022. URL www.h:ttps/....m.... /ttp//aavvvv/////////////////we wwtg///s////...oo.//// 05991. htt..ttpss//aai..ooo///////////
10

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Sun, Z., Fang, Y., Wu, T., Zhang, P., Zang, Y., Kong, S., Xiong, Y., Lin, D., and Wang, Alpha-clip: A clip model focusing wherever you want. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Zeng, G., Zhang, Y., Wei, J., Yang, D., Zhang, P., Gao, Y., Qin, X., and Zhou, Y. Focus, distinguish, and prompt: Unleashing CLIP for efficient and flexible scene text re- trieval. In ACM MM, pp.
2024a.
Zeng. W. Shu, Y., Li, Z., Yang, D., and Zhou, Y. TexICtrl: Diffusion-based scene text editing with prior guidance control. NeurIPS, 1111388669-118994 2024b.
Wang, H., Bai, X., Yang, M., Zhu, S., Wang. J., and Liu, W. Scene text retrieval via joint text detection and similarity learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4558- 4567, 2021.
Zhang, R., Zhou, Y., Jiang, Q., Song, Q., Li, N., Zhou, K., Wang, L., Wang, D., Liao, M., Yang. M., et al. Icdar 2019 robust reading challenge on reading chinese text on sign- board. In Proceedings of the International Conference Document Analysis and Recognition (ICDAR), 2019.
Wang, K. and Belongie. S. Word spotting in the wild. In Proceedings of the European Conference on Computer Vision (ECCV), 2010.
Zhang, Y., Liu, C., Wei, J., Yang, X., Zhou, Y., Ma, C., and Ji, X. Linguistics-aware masked image modeling for self- supervisee scene text recognition. In CVPR, 2025.
Wang, W., Zhou, Y., Lv, J., Wu, D., Zhao, G., Jiang, N., and Wang, W. TPSNet: Reverse thinking of thin plate splines for arbitrary shape scene text representation. In 2022.
Zhou, C., Loy, C. C., and Dai, B. Extract free dense labels from clip. In Proceedings of the European Conference on Computer Vision (ECCV), 2022a.
ACM MM, pp.
Wei, J., Zhang, Y., Zhou, Y., Zeng, G. Qiao, Z., Guo, Y., Wu, H., Wang. H., and Wang, W. TextBlock: Towards scene text spotting without fine-grained detection. In ACM MM, pp. 5892-5902, 2022.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Conditional prompt learning for vision-language models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Wen, L., Wang. Y., Zhang. D., and Chen, G. Visual matching is enough for scene text retrieval. In Proceedings of the Sixteenth ACM International Conference Web Search and Data Mining (WSDM), 2023.
Wilkinson, T. and Brun, A. Semantic and verbatim word spotting using deep neural networks. In Proceedings of the International Conference Frontiers in Handwriting Recognition (ICFHR), 2016.
Xing, Y., Kang, J., Xiao, A., Nie, J., Shao, L., and Lu, S. Bridging semantic gaps for segmentation. CORR, 2023.
semantic
Yang. A., Pan, J., Lin, J., Men, R., Zhang, Y., Zhou, J., and Zhou, C. Chinese clip: Contrastive vision-language pretraining in chinese, 2023. URL https:://///... //rxiv.. org/ abs/ 2211. 01335.
Yang, X., Qiao, Z., and Zhou, Y. IPAD: Iterative, parallel, and diffusion-basee network for scene text recognition. IJCV, pp. 1-21, 2025.
Yu, W., Liu, Y., Hua, W., Jiang, D., Ren, B., and Bai, X. Turning a clip model into a scene text detector. Pro- ceedings of the IEEF/CVE Conference on Computer Vi- sion and Pattern Recognition (CVPR), 2023.
Zeng, G., Zhang, Y., Zhou, Y., Yang, X., Jiang, N., Zhao, G., Wang, W., and Yin, X.-C. Beyond OCR + VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa. Pattern Recognition, 138:109337, 2023.

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
A. Analysis of DL-CSVTR
This section supplements the statistical analysis of CSVTR the definition of query terms our proposed DL-CSVTR.
A.1. Analysis of text layout distribution in CSVTR
We have analyzed the four representations of query terms in CSVTR. The statistics are based on the image as the basic unit, with priority given to horizontal, then partial, then cross-line, and finally vertical layouts. For example, if the visual representation of query term in an image exists both horizontally and cross-line, the image is classified having horizontal layout. The statistical results are shown in Figure 8.
Partial 3%
Cross-Line 1%
Horizontal
93%
other 7%
Vertical
3%
Figure 8. Distribution of text layouts for visual representations of query CSVTR
The statistical results clearly show that the visual layout of most query terms horizontal, which does not reflect the challenges that diverse text layouts pose to scene text retrieval tasks.
A.2. Defining the query terms in DL-CSVTR
The query terms need to appear in a variety of scene images, so they must be common, figh-frequency words. the Chinese context, these are mostly Chinese trademark names, idioms, or noun concepts with specific semantic meanings. Based on our analysis and the definition of query terms in CSVTR, we designed query terms for the three types of text layout benchmarks in DL-CSVTR, shown in Figure 2.
For each defined query term, used common image search engines and manually screened images to find visual representations of the query terms that match the specific text layouts, ensuring that the three layouts remain distinct in our DL-CSVTR.
A.3. Sample visualization in DL-CSVTR
We visualize data from the three benchmarks in DL-CSVTR to provide clearer understanding of the DL-CSVTR dataset. The visualization results are presented in Figure 10.
In DL-CSVTR-V V, the visual representation the query terms is in a vertical layout, while DL-CSVTR-CL. and DL- CSVTR-P, the representation spans rows or columns and connected other characters.
12

---

Beyood Cropped Regionns New Benchmate and Corresponaing Baseline for Chinese Scene Text Retrieval in Diverse Layouts
a DL -L-CSVTRV
DL-CSVTR-CL Cross-Lime =ll
(c) DL-CCVVRR
(b) DI-CSVTR-CL
Vortical
Partàal
NAIXU
2
HFiHHJルSS...
(P1)
FERUEEI
E
Mズ DEL  /
SURKZE
FIIZEE
デテラビス
AHDKKERA
|
fix, शह
お
-
サ
i
8 #IIZ
E
Ti 9. Distribution of text layouts for visual ampresenttations of query terms in CSVTR
Figure
A.4. Aemplemenntarrn ssamples of DL-CSVTR
a the author or We provide a portion of the DIRCS dataset, ensuring that the images do not reveal Information about
associated entities. The dataset includes the following:
1. CLLCSVTR V: This folder contains a subset of vertically arranged data, where the visual i instances of the query words
appeeat a in vertical layout within the images.
2. CLLCSTRR This folder contains a subset of cross-row layout data, where the visual i instances of the query words pords
appear a cross-row or cross-column cross-column layout within the Images.
i instances of the query words appeat
3. DL-CSVTR-P. This folder contains a subset of partial layout data, where the visual Connected to other text within the image..
13

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Algorithm 1 Training Triplet Generation
OCR dataset with images } and their text annotations {T where is the image index and k is the text line index.
Input:
Output: Set of triplets {[T; Gik, } for training
1: for each image I; in the dataset do
2: Initialize an empty set of triplets T
3: for each text line annotation Tik in I; do
4: Initialize grayscale image Gas of the same size as with all pixel values set to
5: Get the polygonal coordinates Pa of the text line Tik
6: Set the pixel values within the polygon P.k in Gi to 255
7: Add the triplet (I,, Gike Tik) to T
8: end for
9: end for
10: Store the triplets T
11: return All stored triplets {(I, Giky 3
Algorithm 2 Retrieval bigrams Generation for Multiple Images using PaddleOCR
Input: Set of original images {
Output:
Set of retrieval pairs {(Tz, Ga
1: Initialize empty set of pairs P
2: for each image 1; in the image set { do
3: Use PaddleOCR to perform text detection on image I, obtaining a set of bounding boxes { {Bub)
4: for each bounding box Bi in detection results }Bab} do
5: Initialize grayscale image Ga of the same size as with all pixel values set to
6: Get the polygonal coordinates Pa of the bounding box Bis
7: Set the pixel values within the polygon P.k. in G to 255
8: Add the pair (1,, G(k) to P
9: end for
10: end for
11: return All generated pairs P
B. Details of the Model
This section provides additional explanations regarding the model details and specific implementation.
B.1. Training triple generation
We generate training triplets based on the text line annotations of the images for both training and testing in the first and second stages. Specifically, the data generation algorithm creates the segmentation map mentioned in the text, based on the text line annotations. This segmentation map is a grayscale image that forms a triplet with the corresponding original image and text. Please refer to the pseudocode provided in Algorithm
B.2. Retrieval bigrams generation
In the retrieval stage, thanks to the standard two-tower model we use, we only need to create a segment map for the images in the gallery to get the bigrams. We use Paddle OCR, an OCR detection model with strong performance and fast inference speed, as the detector. The detector is used to give the location annotations of multiple text regions and generate a segmentmap to obtain candidate bigrams. Please refer to the pseudocode provided in Algorithm
B.3. Supplementation of RAGP
To better understand the processing order of RAGP, we provide pseudocode to illustrate the process. Please refer to Algorithm 3.
14

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Algorithm 3 Random Granularity Alignment Processing
Input: Text inputs Tik, Grayscale images Ga
Parameter:
3. a, 9
Output: Processed text inputs and grayscale images
1: for each text input Ti do
2: Random Expand:
3: Find Tpp whose centroid closest to Tak
4: With probability a, merge Tak and Tip
5: Random Mask:
6: if length(Tax) then
7: With probability B. delete m characters from Tik ensuring length 2 2
8: end if
9: end for
10: for each grayscale image Gi do
Random Splicing:
Find Gip whose centroid is closest to Gik
13: With probability 0, merge Ga and Gip
14:
end for
The pseudocode clearly illustrates the process the RAGP we proposed. In the actual code implementation, we accomplish this random processing algorithm using the getitem function and other custom-defined functions when creating the dataset.
C. Visualization of Retrieval Results.
C.1. Retrieval visualization on DL-CSVTR
To intuitively analyze the retrieval capabilities of STR-CLIP and previous methods on Chinese text retrieval with diverse layouts, visualization results for three different text layout benchmarks from DL-CSVTR are presented in Figure 11.
Under the DL-CSVTR-V benchmark, as shown in Figure 11()), previous methods based on cropping text regions discard visual information outside the text region, hindering model's ability to understand vertically arranged text. For instance, in positive examples, we observed that vertical text in the scene is often associated with surrounding visual elements, such as vertical billboards and architectural styles, which are crucial for the model's comprehension of vertical layouts. Our method, STR-CLID achieves the best retrieval results by understanding text information and related visual elements from full-image perspective. The AP comparison for each query term is shown in the figure 12.
Under the DL-CSVTR-CL benchmark, as shown in Figure 11(b), our proposed model benefits from breaking the limitations of the cropped text region paradigm. Even if the text region that guides the model to perceive only contains part of the query words, our full-image information perception and multi- granularity perception guided by the text region can solve this problem well. The AP comparison for each query term is shown in the figure 13.
Under the DI-CSVTR-P benchmark, shown in Figure 1 Ic()), our proposed model outperforms the previous single- granularity matching paradigm by enhancing the perception of specific elements within the text region. The AP comparison for each query term is shown in the figure 14.
C.2. Retrieval visualization on CSVTR
The AP comparison for each query term is shown in the figure 15.
D. Supplementary of Experiments
D.I. Details baseline reproduction
We modified the open-source code of TDSL (Wang et al., 2021) and Luo et al (Luo et al., 2024) which represent the most advanced scene text retrieval models for cross-modal embedding and visual embedding, respectively. Specifically, for TDSL.
15

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
replaced the language and visual encoders with the pre-trained Chinese CLIP encoder and fine-tuned it using cropped text regions and annotations. For VSTR, we replaced the dual-end visual encoder with the visual encoder from Chinese CLIP and trained it using cropped text regions and annotated text instance images generated from annotations. After replacing the encoder, both models demonstrated stronger retrieval ability on the previous CSVTR benchmark.
D.2. Supplement of model code
We provide the model code and data processing scripts used in our work, along with some data to illustrate our model training process. We have ensured that all content potentially leaking personal information has been removed. Specifically, the provided code and related data include the following components:
1. CSTR-CLIP-StgeeLpp. This file contains the model code and dataset loading script for the first stage of training CSTR-CLIP.
2. CSTR-CI.IP-Stage..p: This file contains the model code and dataset loading seript for the second stage of training CSTR-CLIP. The RAGP algorithm we designed incorporated into the dataset design.
3. train-img: maps.
This folder contains a portion of the data used for training, including original images and segmentation
16

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Benchmark
Query
Example
(a)
DL-CSVTR-V
"M
McDonald's
"
OOF
(b)
DL-CSVTR-CL
"Mi
CCTT COPFEE
"
AVIT
SE AB SR S
(c)
DI.CSSTT-PP
M
Figure 10. Sample presentation of three benchmarks in DL-CSVTR, Best viewed in zoom.
17

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
#
Benchmark
Query
Method
Retrieval Results
Visual Embeding
Cross Model Embeding
(a)
DL-CSVTR-V
CSTR-CLIP
Visual Embeding
WISSS
JR DN
FR
Cross Model Embeding
RO
(b)
DL-CSVTR-CL
"MAL
CSTR-CLIP
Visual Embeding
Cross Model Embeding
(c)
DI-CSVTR-P
"
IXI EXLEOOOD
CSTR-CLIP
Figure 11.. Visualization of retrieval results. (a) example DI-CSVTR-V henchmark, which rank @ 1-5 retrieval results provided. (b) An example on DI-CSVTR-CL. benchmark, in which rank @ 1-5 retrieval results are provided. (c) An example on
DL-CSVTR-P benchmark, in which rank @ 1-5 retrieval results are provided. The correct results are highlighted in green while the incorrect ones are highlighted in red. Best viewed in zoom.
18

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Comparison of Different Models DL-CSVIR-V
Cross- Midel Embedding
Visual Embedding
0.8
BIRAS
BBR DR BUUBDS
AURU
4:30
Quaries
Figure 12. Comparison of AP of each query word under the DL-CSVTR-V benchmark.
Comparison Different Models DL-CSVTR-CL
Creas-Model Embedding Visuol Embedding CSTR-CUP
0.0
7U57
R
MB-RR
WIRKB
ROKKRE
06en
Queries
Figure 13. Comparison of AP of each query word under the DL-CSVTR-CL benchmark.
19

---

Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Compatison of Difierent Models- DL-CSVTR-P
Cross-Model Embeddiig
Visual Embedding
CSTR-CUP
0.6
0.4
0.2
490
1010
2150
1505
Queriies
Figure 14. Comparison of AP of each query word under the DL-CSVTR-P benchmark.
Comparison
10-
Cross-Model Embedding
Embedding CSTR-CUP
0.8
0.2
Queries
Figure 15. Comparison of AP of cach query word under the CSVTR benchmark.
20


Reviews:
[{"review_id": "PqPXlVn2p6", "reviewer": "Reviewer_osp1", "summary": "In this paper, the authors aim to solve the problem of Chinese scene text retrieval in complex and diverse layouts. They first establish the DL-CSVTR benchmark including vertical, cross-line and partial alignments. In addition, the authors propose CSTR-CLIP method which integrates global visual information with multi-granularity alignment training. The experiments are conducted on both previous benchmark and the proposed DL-CSVTR demonstrating the proposed method CSTR-CLIP outperforms previous SOTA model.\n\n## update after rebuttal\nI maintain my score after reading rebuttal", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1**\n\nIn Figure 2, we can conclude that horizontal text occupies 92.62%. It is not clear how the oriented text is classified – is it classified as horizontal or vertical?\n\n**Response 1**\n\nWe appreciate the reviewer’s observation. To clarify, we classified the visual representation of query text in the CSVTR dataset based on manual verification of their layout in the images. Specifically, when both horizontal and vertical text appear in the same image, we categorize the query text based on the priority of visibility and completeness. The priority order for classification is as follows: cross-line > partial > vertical > horizontal. In other words, if both horizontal and vertical text are present in the same image, the text would be classified as vertical. This approach ensures a consistent and logical classification across all images, accounting for different layout variations.\n\n**Question 2**\n\nOther than vertical, cross-line, and partial text retrieval, is there any other type of retrieval task that has not been well addressed?\n\n**Response 2**\n\nIn terms of Chinese text layouts, there is another layout type that we believe requires further attention: dispersed layout. For instance, a query term like \"ICML\" could appear in an image as \"International Conference on Machine Learning\", where the characters are spaced out. This scenario represents a potential area of improvement, and we are working on solutions to address this issue.\n\nFurthermore, we believe two additional challenges in scene text retrieval should be explored in future work: 1) Retrieval of text with specific attributes, such as layout and color; 2) Retrieval of text in relation to visual elements, for example, \"a blue building with the text 'xxx' on it.\" We see these as promising directions for future advancements in scene text retrieval. We will discuss it in the future work in our camera-ready version.", "claims_and_evidence": "The claims made in this paper include the specific characteristics of Chinese scene text retrieval, which is evident since Chinese is different from English obviously.", "methods_and_evaluation_criteria": "They are reasonable because Chinese has more vertical textlines than English, and cross-line and partial retrieval are useful because Chinese textline is composed of characters while English textline is composed of words.", "theoretical_claims": "It seems there is no proof for theoretical claim involved.", "experimental_designs_or_analyses": "The performance on CSVTR benchmark, DL-CSVTR benchmark and ablation study are all checked, and the results sound solid.", "relation_to_broader_scientific_literature": "t pushes the general CLIP model to extensive text retrieval task with sophisticated design.", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "Strengths:\n\n(1) In the paper, the authors find that Chinese scene text retrieval is different from English in that Chinese text has many layouts, so they do not only transfer English methods to Chinese scenario but design new retrieval tasks including vertical, cross-line and partial retrieval where English retrieval seldom involves. This is significant since Chinese text together with other similar texts are also used all over this world.\n\n(2) The proposed CSTR-CLIP method further pushes the CLIP into Chinese scene text retrieval task, verifying that CLIP has the potential in this task though it is the by-product. The CSTR-CLIP performs well on Chinese benchmarks.\n\n(3) The experiments are enough to verify the effectiveness of the proposed method, and many extra experimental results are supplied in the appendix material.\n\nWeaknesses:\n\n(1) In figure 2, we can conclude that Horizontal text occupies 92.62%. I is not clear about how is the oriented text classified? To horizontal or vertical?\n\n(2) Other than vertical, cross-line and partial text retrieval, is there any other type of retrieval task that has not been well addressed?", "other_comments_or_suggestions": "see the weakness part.", "questions_for_authors": "see the weakness part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NO"}}, {"review_id": "E6oRBVyeFm", "reviewer": "Reviewer_UNMy", "summary": "This paper addresses the limitations of existing Chinese scene text retrieval methods, which inherit the solution for English scene text retrieval and fail to achieve satisfactory performance in Chinese scene text retrieval. Therefore, the authors first introduce DL-CSVTR, a new benchmark featuring vertical, cross-line, and partial text layouts for more realistic assessments. Then, they propose CSTR-CLIP. CSTR-CLIP applies a two-stage training process to overcome previous limitations, such as the exclusion of visual features outside the text region and reliance on single-granularity alignment, thereby enabling the model to effectively handle diverse text layouts. Experimental results show that CSTR-CLIP outperforms existing methods significantly on both the standard CSVTR dataset and the new DL-CSVTR benchmark, effectively handling varied text layouts.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1**\n\nIn Table 2, we found that CSTR-CLIP has significantly improved performance in various challenge scenarios. However, the basic horizontal cases are not reported together. Can the author provide the corresponding performance results?\n\n**Response 1**\n\nWe appreciate the reviewer’s comment. We have taken CSVTR as the baseline for horizontal retrieval, as most query text in the dataset (as shown in Figure 2) predominantly follows a horizontal layout. Therefore, the performance of CSTR-CLIP in horizontal retrieval scenarios is essentially reflected in the CSVTR baseline results. For further reference, the results in Table 1 of the paper can provide additional insights into the performance for horizontal retrieval.\n\nTo elaborate, we manually categorized the visual representations of the query text in the CSVTR dataset based on their corresponding layouts. In cases where a single image contained multiple orientations, such as both horizontal and vertical text, we applied a priority-based classification process. Specifically, if both orientations were fully visible and intact, we followed this priority: cross-line > partial > vertical > horizontal. For instance, if both horizontal and vertical layouts appeared in the same image, we classified the layout as vertical. After conducting this analysis, we found that 92.62% of the data had query words with a horizontal visual representation in the image. Consequently, we have used CSVTR as the baseline for horizontal layout retrieval and did not collect additional horizontal layout data in the DL-CSVTR dataset.\n\n**Question 2**\n\nAlthough this work is designed for Chinese scene text retrieval, all the problems mentioned are also challenges for the English scene text retrieval task. Can the author provide the performance results of CSTR-CLIP on English scene text retrieval datasets? I think it will be helpful to evaluate its improvement upon recent works on more general datasets.\n\n**Response 2**\n\nThank you for raising this important point. We agree that the challenges discussed in our work are indeed also relevant to English scene text retrieval. In response, we trained CSTR-CLIP using the en-clip model with the same processing approach and evaluated its performance on English scene text retrieval datasets.\n\nRegarding the data, we followed the methodology used in SynthText-900K for the English corpus and generated a dataset of 300k images, matching the scale of the pre-trained data used for the Chinese scene text retrieval. For real-world data in the second stage, we used the English subset of the MLT dataset as the training set, aligning it with previous work in English scene text retrieval.\n\nIn terms of model design, we retained the two-stage framework of CSTR-CLIP, replacing the cn-clip with en-clip as the initialization model. We chose the IIIT Scene Text Retrieval dataset as the benchmark for English scene text retrieval, using mean Average Precision (mAP) as the evaluation metric to compare our method's performance in English scene text retrieval. The results are shown in the table below:\n \n| **Method**          | **mAP** |\n|-------------------------------|---------|\n| TDSL                | 77.09   |\n| VSTR                | 77.40   |\n| Luo et al           | 82.15   |\n| Ours (Stage 1)      | 83.17   |\n| Ours (Stage 2)      | ***86.75***   |\n\n\nCSTR-CLIP also demonstrated excellent performance in the English scene text retrieval context. However, we observed that cross-line and partial layouts are less common in the English scene. This was evident in the visual representation of query words in the IIIT Scene Text Retrieval dataset, which showed a similar trend. We believe the key to performance improvement lies in the retention of full-image information and the guidance provided by region suggestions. This is particularly important because many images in the IIIT dataset have lower clarity, and previous crop-based methods struggle to effectively retrieve text from such images. The second-stage improvement likely comes from fine-tuning on real-world data. Therefore, the \"Beyond Cropped Regions\" paradigm is indeed beneficial for English scene text retrieval as well. We also plan to open-source CSTR-CLIP for English scene text retrieval in the near future.", "claims_and_evidence": "Yes, I think the claims in this submission are supported by their experiments and discussions.", "methods_and_evaluation_criteria": "Yes, I think the proposed method and evaluation criteria are proper and make sense for the problem.", "theoretical_claims": "Yes, I have checked the correctness of the proofs for the theoretical claims and found no issues.", "experimental_designs_or_analyses": "Yes, I reviewed the experimental design and related analyses and found them reasonable and valid.", "relation_to_broader_scientific_literature": "This work focuses on Chinese scene text retrieval and proposes complex and variable datasets. Therefore, I think it makes sense for a border impact in the whole scene text retrieval community. Besides the dataset, the author also introduces a method named CSTR-CLIP, which uses full image information with guided attention instead of previous crop-detection processing. I think it is somewhat reasonable and interesting.", "essential_references_not_discussed": "I think this paper adequately cites and discusses the essential related works that are necessary for understanding the context of its key contributions.", "other_strengths_and_weaknesses": "Strengths:\n- It is nice to see this work proposes a new dataset DL-CSVTR. This dataset is interesting and contains a large number of challenges from real-world scenarios.\n- Performance of this work is quite well. Especially in those challenge cases such as vertical, cross-line, and partial alignments.", "other_comments_or_suggestions": "I have no more extra comments.", "questions_for_authors": "1. In Table 2, we found that CSTR-CLIP has significantly improved performance in various challenge scenarios. However, the basic horizontal cases are not reported together. Can the author provide the provide the corresponding performance results?\n\n2. Although this work is designed for Chinese scene text retrieval. However, all the problems mentioned are also challenges for the English scene text retrieval task. Therefore, can the author provide the performance results of CSTR-CLIP on English scene text retrieval datasets? I think it will be helpful to evaluate its improvement upon recent works on more general datasets.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k9pS1S7AU8", "reviewer": "Reviewer_QR6X", "summary": "This paper focuses on Chinese scene text retrieval, which aims to extend previous English scene text retrieval to Chinese. The authors establish a Diversified Layout benchmark for Chinese Street View Text Retrieval (DL-CSVTR) to assess retrieval performance across different text layouts. They also propose Chinese Scene Text Retrieval CLIP (CSTR-CLIP), a new model integrating global visual information and multi-granularity alignment training. Experiments on existing benchmarks show that CSTR-CLIP achieves an 18.82% accuracy improvement over the previous SOTA model and has a faster inference speed. Analysis of DL-CSVTR validates its superiority in handling diverse text layouts.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1**\n\nTab.1, why is CSTR-CLIP much faster than all other methods except CLIP? This should be explained in detail.\n\n**Response 1**\n\nThank you for the valuable comment. The faster performance of CSTR-CLIP compared to other methods can be attributed to the simplified nature of our approach. Previous methods based on visual embedding techniques such as [1][2], require additional computational effort to crop text regions, and apply model-based style processing to convert those regions into a standardized format for easier matching. Furthermore, these methods also required extra steps to transform the query into an image format that could match the cropped text regions, leading to a significant loss in inference speed. Additionally, earlier approaches based on cross-modal embeddings [3] involved extra costs for designing matching templates and performing region cropping, which further slowed down the inference process.\n\nIn contrast, CSTR-CLIP benefits from a direct cross-modal matching approach, eliminating the need for extra style transformations, rendering, or template-based matching constraints. This streamlined process allows for much faster inference speed. We will clarify this explanation in the manuscript to provide a deeper understanding of the performance improvements.\n\n**Question 2**\n\nFig.7, what is the application for \"Interactive Region-Specified Scene Text Retrieval\"? If the user can give the mask, why do they not only input the mask region as a separate image?\n\n**Response 2**\n\nWe appreciate your thoughtful question. \"Interactive Region-Specified Scene Text Retrieval\" is designed to refine the granularity of user searches. When users need to search for images containing specific text within their image library and have some recollection of the region where the text appears, they can provide a region suggestion. By focusing on the region of interest, this enhances the search results, improving accuracy and recall in retrieval tasks. However, since the region specified by the user may not perfectly match, directly cropping the image could lead to information loss and additional computational costs. Thanks to the design and training paradigm of CSTR-CLIP, the model does not limit itself to the given region but also responds to surrounding areas within the suggested region, as illustrated in Figure 6. Therefore, CSTR-CLIP can improve retrieval accuracy and recall by considering the user's region suggestion and query text, all while avoiding information loss.\n\n[1]Visual and semantic guided scene text retrieval\n\n[2]Visual Matching is Enough for Scene Text Retrieval\n\n[3] Scene text retrieval via joint text detection and similarity learning.", "claims_and_evidence": "Yes, confirmed.", "methods_and_evaluation_criteria": "Yes, confirmed.", "theoretical_claims": "This submission does not involve proof of theory.", "experimental_designs_or_analyses": "Yes. I have checked all the experimental results, including the comparison with previous SOTA and the ablation of modules of all stages. The experiments are sound and extensive and are superior to previous works.", "relation_to_broader_scientific_literature": "Extending text retrieval task to Chinese scenario. It could identify limitations in prior research.", "essential_references_not_discussed": "I hold there are relatively sufficient essential references discussed.", "other_strengths_and_weaknesses": "Strengths:\n1) This paper constructs a Chinese scene text retrieval dataset, which includes various layouts, and can support vertical/cross-line/partial text retrieval that English text retrieval seldom encounters.\n2) They propose CSTR-CLIP method for Chinese scene text retrieval, which relieves text detection needs and enhances perception flexibility through multi-granularity alignment.\n3) Extensive experiments show that the proposed method achieves superior performance on both the previous Chinese scene text retrieval dataset and the proposed DL-CSVTR dataset.\n\nWeaknesses:\n1) Tab.1, why CSTR-CLIP is much faster than all other methods except CLIP should be explained in detail.\n2) Fig.7, what is the application for “Interactive Region-Specified Scene Text Retrieval”? If the user can give the mask, why do they not only input the mask region as a separate image?", "other_comments_or_suggestions": "Please refer to the above comments.", "questions_for_authors": "Please refer to the above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZG2Xlzjp7c", "reviewer": "Reviewer_7i8U", "summary": "The paper addresses Chinese scene text retrieval challenges, focusing on the complex layouts of Chinese text in real-world scenes. Current approaches that adapt English text retrieval methods to Chinese contexts show limited performance. The authors introduce DL-CSVTR, a benchmark for evaluating Chinese text retrieval across diverse layouts including vertical, cross-line, and partial alignments --- addressing limitations in existing datasets that primarily feature horizontal text. They also propose CSTR-CLIP, a novel model that moves beyond cropped text regions by employing a two-stage training approach. A key innovation is the Random Alignment Granularity Processing module that improves perception of text elements both within and around text regions. Experiments show CSTR-CLIP outperforms previous methods on both existing benchmarks and the new DL-CSVTR benchmark, particularly for challenging text arrangements.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comment, and we would like to clarify these questions according to subjects.\n\n**Question about DL-CSVTR datasets**\n\n1. Claims And Evidence's Para 2\n\n   We ensured that the process involved three annotators, with one main annotator overseeing the data quality and consistency. The main annotator was responsible for validating the layout classification, image quality, and privacy considerations of the data submitted by the other two annotators. Additionally, the data provided by the main annotator was shared with the other annotators for reference. After annotating six query words, all three annotators would engage in discussions to identify and resolve any inconsistencies, thus ensuring alignment in the annotations. Regarding layout categories, we employed manual quality control to ensure that the visual representation of query words in images strictly adhered to the corresponding layout categories. We ensured that no additional layout formats for the same query word interfered with the classification.\n\n2. Methods And Evaluation Criteria's Para 3\n\n   DL-CSVTR refers to the scale of scene text retrieval benchmarks in both Chinese and English scene text retrieval tasks, such as CSVTR, IIITSceneTextRetrieval, and StreetViewText. This approach ensures that the dataset is appropriately aligned with existing benchmarks, making it well-suited for evaluating scene text retrieval performance.\n\n3. Methods And Evaluation Criteria's Para 2 \n\n   The 89 query words, sourced from real street scenes including trademarks and phrases, were manually selected and clustered by annotators based on layout requirements in large-scale images. Data collection preceded model design, as our prior research identified gaps in handling specific Chinese text layouts. To address this, we created DL-CSVTR using the CSVTR methodology, ensuring the dataset's relevance and authenticity with real-world street view scenes.\n\n**Question about CSTR-CLIP model**\n\n1. Weaknesses 1, 3 & Experimental Designs Or Analyses's Para 3\n\n   RAGP relaxes the strict constraints of location-based suggestions. After the first training stage, the model matches regions indicated by Textpos Conv. However, layouts like cross-line and partial layouts may only appear partially within the detection region. RAGP mitigates this by adjusting the alignment of the suggestion region with the query, improving performance for layouts that don’t fully align, especially for cross-line and partial layouts.\n\n   The modest improvement with CSVTR is because RAGP addresses location constraints, which are less relevant for horizontally aligned query words, as shown in Figure 2.\n\n   We also appreciate the reviewer’s suggestion to include the pseudocode for RAGP. We will provide a more detailed description of the RAGP algorithm in the camera-ready paper.\n\n2. Other Comments Or Suggestions 1\n\n   we adopted the same kernel size as the first convolutional layer of CLIP's image data preprocessing to maintain compatibility with subsequent fusion.\n\n3. Other Comments Or Suggestions 2\n\n   The variation in retrieval accuracy across different query words is most evident in cross-line and partial layouts, due to differences in the spatial distances between query word parts in cross-line layouts and the varying paragraph lengths in partial layouts. These issues are closely related to RAGP, which guides the model to focus on the visual features around the suggested region. The effectiveness of RAGP is influenced by hyperparameters α, β, and θ, which control the receptive field size. We conducted experiments and visualized bad cases to determine the optimal combination of these parameters during training.\n\n**Question about experiment design**\n\n1. Weaknesses 4\n\n   We chose GOT as a transformer-based approach, using spotting-related instructions. Specifically, edit distances between a query word and the spotted words from scene images are used for text-based image retrieval.\n\n   | Method |  CSVTR | DL-CSVTR-V |  DL-CSVTR-CL |  DL-CSVTR-P |\n   |-----|-----|-----|-----|-----|\n   | GOT |86.56| ***84.91*** |  59.47 | 55.78 |\n   | Ours |***88.57***| 84.44 | ***65.56*** | ***61.83*** |\n\n2. Experimental Designs Or Analyses Para 3\n\n   Recall is suitable for single-target queries, but in DL-CSVTR, a query word can correspond to multiple targets, making it unsuitable. mAP, which has been used as the sole metric for accuracy in previous scene text retrieval works, is therefore employed in this study.\n\n3. Other Comments Or Suggestions 3\n\n   Please refer to Reviewer QR6X's response 1.\n\n4. Weaknesses 2\n\n   Errors mainly occur in cross-line layouts with large gaps between query word parts and in partial layouts where the query word occupies a small portion of the string. These issues contribute to bias in errors. RAGP with global feature fusion improved performance, but for challenging samples, further attention to semantically related elements outside the suggested region may be needed.", "claims_and_evidence": "The paper presents convincing evidence for some claims but falls short in others. The performance improvements of CSTR-CLIP are well-documented in Tables 1 and 2, and the ablation studies in Table 3 effectively demonstrate component contributions. \n\nI find the construction and validation of the DL-CSVTR benchmark problematic. The paper mentions three annotators collected the data but provides minimal details on annotation protocols or quality assurance. For a benchmark paper, this is a significant weakness. Were there inter-annotator agreement measurements? What specific criteria determined layout categories? The dataset size (2,070 images) seems modest for a benchmark intended to evaluate real-world performance.\n\nThe causal analysis linking model components to performance gains is weak. For instance, when discussing the RAGP module, the paper shows it helps in cross-line and partial layouts (Table 3, rows 5 vs 6), but doesn't adequately explain the mechanism. Why does random granularity alignment specifically help with these layouts? The visualization in Figure 6 is interesting but doesn't fully connect to quantitative results.\n\nThe paper lacks error analysis - when does CSTR-CLIP fail and why? The performance on individual queries varies dramatically (visible in Figures 12-14 in the supplementary material), but this variation isn't analyzed in the main text. Without understanding failure modes, it's difficult to fully assess the model's robustness.", "methods_and_evaluation_criteria": "The proposed methods align well with the unique challenges of Chinese scene text retrieval. Moving beyond cropped regions is particularly sensible, as Chinese characters often appear in complex spatial arrangements that traditional bounding box approaches can't handle effectively. The two-stage training process reflects a thoughtful consideration of what's actually needed: both OCR capabilities and layout understanding.\n\nI'm less convinced about some aspects of the evaluation framework. While creating a dedicated benchmark for diverse layouts addresses a real gap, the construction feels somewhat ad hoc. The authors manually collected images for different layout categories, but provided little justification for why these specific 89 query terms were chosen or how well they represent real-world retrieval scenarios. A more systematic approach might have started with an analysis of query distributions from actual user data.\n\nThe benchmark's size (around 2,000 images) strikes me as minimal. For comparison, standard object detection benchmarks typically include tens of thousands of images. This raises questions about whether performance gains would generalize to larger, more diverse datasets.\n\nOne methodological strength is the comparison with both CLIP baselines and prior specialized approaches. This shows the contribution beyond just leveraging a strong pretrained model. The speed-accuracy tradeoffs are also reasonably explored, though real-world deployability considerations could have been discussed more thoroughly.\n\nIn summary, while the methods are well-matched to the problem, the evaluation criteria would benefit from more rigorous benchmark construction and validation.", "theoretical_claims": "The paper is primarily empirical in nature and does not present formal mathematical proofs or theoretical guarantees that require verification. The contributions are algorithmic and experimental rather than theoretical.\n\nThe authors do make some informal claims about why their approach works, particularly regarding the limitations of cropped-region paradigms and the benefits of multi-granularity alignment, but these are supported through ablation studies and experimental results rather than formal proofs.\n\nThe paper's technical foundation largely builds on the existing CLIP architecture with modifications specific to the Chinese scene text retrieval task. The authors explain their algorithmic contributions (like the Random Alignment Granularity Processing module) but don't provide theoretical convergence guarantees or complexity analyses that would typically accompany theoretical papers.\n\nGiven the applied nature of the work, the absence of formal proofs is not necessarily a weakness. However, a stronger theoretical analysis of why multi-granularity alignment specifically addresses the challenges of diverse Chinese text layouts could have strengthened the paper's contributions beyond empirical results.", "experimental_designs_or_analyses": "I examined several aspects of the experimental design and found some methodological issues:\n\nThe DL-CSVTR benchmark construction lacks statistical rigor. The authors use three annotators, but don't report inter-annotator agreement or formalized criteria for categorizing layouts. This raises questions about the benchmark's reliability and reproducibility.\n\nFor evaluation metrics, they rely solely on mean Average Precision (mAP), which is standard but insufficient alone. Given their focus on diverse text layouts, layout-specific metrics capturing the unique challenges of vertical or cross-line text would strengthen their analysis.\n\nThe ablation study (Table 3) is reasonably designed to isolate component contributions, but lacks error bars or significance testing. With performance differences sometimes being modest (like 88.41% vs 88.57% on CSVTR), it's hard to assess whether improvements are meaningful or statistical noise.\n\nThe visualization analysis (Figure 6) offers qualitative insights but feels cherry-picked. A more systematic visualization approach across different query types would better support claims about the model's perceptual abilities.\n\nThe baseline implementations warrant scrutiny. The authors replace backbones in previous methods with CLIP for \"fair comparison,\" but this modification fundamentally changes those methods. I question whether these are still valid representations of the original approaches or essentially new hybrid models.", "relation_to_broader_scientific_literature": "The paper makes two significant contributions that connect to several research threads:\n\nThe DL-CSVTR benchmark extends previous scene text retrieval datasets (like IIIT and CSVTR), addressing their notable bias toward horizontal text layouts. While prior benchmarks established evaluation frameworks, they failed to capture layout diversity in real Chinese text.\n\nCSTR-CLIP relates to recent advances in guided attention for vision-language models. Similar to MaskCLIP and Red-Circle's approach to regional attention, it directs CLIP's focus while preserving global context. However, its full-image approach represents a departure from the dominant cropped-region paradigm established in Mishra's 2013 work and continued through Gomez (2018) and Wang (2021).\n\nThe multi-granularity alignment concept evolved from cross-modal embedding work by Luo and Wen, but introduces flexibility previous approaches lacked.\n\nA notable gap is limited engagement with transformer-based OCR systems that have shown promise for complex layouts. By positioning their work primarily within retrieval rather than OCR literature, the authors somewhat constrain their conceptual framework.", "essential_references_not_discussed": "Based on my knowledge, I did not identify any major omissions in the related work.", "other_strengths_and_weaknesses": "### Strengths:\n1. The paper addresses a practical problem in Chinese scene text retrieval that differs significantly from English text retrieval challenges. The authors effectively identify limitations in applying English-centric methods to Chinese texts with diverse layouts.\n\n2. Their technical approach cleverly combines CLIP's visual understanding with layout-specific guidance. Rather than merely cropping text regions, they preserve global context while directing attention to text areas.\n\n3. The two-stage training process shows careful consideration of the problem's unique requirements. The first stage focuses on core OCR abilities while the second enables handling complex layouts.\n\n### Weaknesses:\n1. The experimental analysis lacks depth in connecting model behavior to performance across different layout types. While ablation studies show component contributions, they don't sufficiently explain why certain components help specific layouts.\n\n2. The performance differences shown in supplementary figures (12-15) reveal considerable variation across query types that deserves more thorough analysis. This would provide more meaningful insights than the aggregate metrics alone.\n\n3. Several technical descriptions need clarification, particularly for the Random Alignment Granularity Processing. The algorithm is central to their approach but described in abstract terms without concrete examples of how it transforms inputs.\n\n4. The paper would benefit from more direct comparison with recent transformer-based approaches that handle complex layouts, situating their work in the broader context of both retrieval and OCR research.", "other_comments_or_suggestions": "1. Section 4 describes $Conv _ {Textpos}$ but doesn't specify its architecture details (kernel size, etc.).\n\n2. The explanation of hyperparameters α, β, and θ in RAGP (page 5) lacks justification for chosen values. How sensitive is performance to these settings?\n\n3. The discussion of inference speed (FPS) in results sections mentions improvements but doesn't analyze computational complexity of different components.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All four reviewers recommend accepting this paper after rebuttal and the final scores of reviewers are 2 Weak accept and 2 Accept. The AC agrees with the reviewers that the paper has made some valuable contributions (i.e., a new benchmark and a new baseline for Chinee scene text retrieval) to the community and deserves to be published in ICML.
Decision: Accept (poster)