GoMatching: A Simple Baseline for Video Text Spotting via Long and Short Term Matching
Authors: Haibin He
Maoyuan Ye
Jing Zhang
Juhua Liu
Bo Du
Dacheng Tao
Venue: NeurIPS 2024 poster (2024)
Keywords: video text spotting
efficient tuning
long and short term matching
Primary area: machine_vision

Abstract: Beyond the text detection and recognition tasks in image text spotting, video text spotting presents an augmented challenge with the inclusion of tracking. While advanced end-to-end trainable methods have shown commendable performance, the pursuit of multi-task optimization may pose the risk of producing sub-optimal outcomes for individual tasks. In this paper, we identify a main bottleneck in the state-of-the-art video text spotter: the limited recognition capability. In response to this issue, we propose to efficiently turn an off-the-shelf query-based image text spotter into a specialist on video and present a simple baseline termed GoMatching, which focuses the training efforts on tracking while maintaining strong recognition performance. To adapt the image text spotter to video datasets, we add a rescoring head to rescore each detected instance's confidence via efficient tuning, leading to a better tracking candidate pool. 
Additionally, we design a long-short term matching module, termed LST-Matcher, to enhance the spotter's tracking capability by integrating both long- and short-term matching results via Transformer. Based on the above simple designs, GoMatching delivers new records on ICDAR15-video, DSText, BOVText, and our proposed novel test set with arbitrary-shaped text termed ArTVideo, which demonstates GoMatching's capability to accommodate general, dense, small, arbitrary-shaped, Chinese and English text scenarios while saving considerable training budgets. The code will be released.

GoMatching: A Simple Baseline for Video Text Spotting via Long and Short Term Matching
Haibin He Maoyuan Ye Jing Zhang",, Juhua Liu Bo Du', Dacheng Tao?
1 School of Computer Science, National Engineering Research Center for Multimedia Software, and Institute of Artificial Intelligence, Wuhan University, China College of Computing & Data Science at Nanyang Technological University (haibinhe, yemaoyuan, liujuhua, dubo edu, com j jingzhang- cvégmail. com, dacheng. tao8gmail. https: /et:://githhn github. com/Hxyz 123/ 2/8//MMatchingg
Abstract
Beyond the text detection and recognition tasks image text spotting, video text spotting presents augmented challenge with the inclusion of tracking. While advanced end-to-end trainable methods have shown commendable performance, the pursuit of multi-task optimization may pose the risk of producing sub-optimal outcomes for individual tasks. this paper, we identify a main bottleneck in the oftte-of-thh-ar video text spotter: the limited recognition capability. In response to this issue, we propose to efficiently turn an off-the-shelf query-basce image text spotter into specialist video and present simple baseline termed GoMatching, which focuses the training efforts on tracking while maintaining strong recognition performance. To adapt the image text spotter to video datasets, add rescoring head to rescore each detected instance's confidence via efficient tuning, leading to a better tracking candidate pool. Additionally, we design a long-short term matching module, termed LST-MMtchhr,, to enhance the spotter's tracking capability by integrating both long- and short-term matching results via Transformer. Based the above simple designs, GoMatching delivers new records on ICDAR15-vide DSText, BOVText, and our proposed novel test with arbitrary-shappe text termed ArTVideo, which demonstrates GoMatching's 's capability to accommodate general, dense, small, arbitrary-shape Chinese and English text scenarios while saving considerable training budgets.
Introduction
Text spotting has received increasing attention due to its various applications, such as video re- trieval [1] and autonomous driving [2]. Recently, numerous image text spotting (ITS) methods [3-6] that simultaneously tackle text detection and recognition, have attained extraordinary accomplishment. In the video realm, video text spotting (VTS) involves tracking task additionally. Although VTS methods [7-12] make significant progress, substantial discrepancy persists when compared to ITS. We observe that the text recognition proficiency of VTS models is far inferior ITS models. To investigate this issue, we compare the state-of-thh-arr (SOTA) VTS model TransDETR [12] and ITS model Deepsolo [5] for image-level text spotting performance on ICDARI5-vide [13] and established Ar''Video (iee,, Arbitrary-shaped Text in Video) test set (Sec.4.1) which comprises about 30% curved text.
As illustrated in Fig. I(a), even when evaluating the image-level spotting performance on the VTS model's 's training set, the F1-score of TransDETR is only comparable the zero-shot performance of
Equal contribution, 1 Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ImsgeLed
6,508 GPU moneryy
Vibraa ore-de))
GoMatching
monory
gowddii credke)
12.66 MOTA better (72.04 vs. 59.38)
TransDETR
197 GPU hours less vs, 200)
15
280
Ne (I Copp Satnen 16 Spriing ill Cop Sepletoon Spa.
GPU Hours
(a)
(b)
Figure 1: (a) 'Gap between Spot. & Det.: the gap between spotting and detection F1-score. As the spotting task involves recognizing the results of the detection process, the detection score is indeed the upper bound of spotting performance. The larger the gap, the poorer the recognition ability, Compared to the ITS model (Deepsolo [5]), the VTS model (TransDETR (12]) presents unsatisfactory image-level text spotting Fi-scores, which lag far behind its detection performance, especially on VrTVideo with curved text. It indicates recogmition capability is a main bottleneck in the VTS model. (b) GoMatching outperforms TransDETR by over 12 MOTA on ICDARI5-video while saving 197 training GPU hours and 10.8GB memory. Notice that since the pre-training strategies and settings vary between TransDETR and GoMatching, the comparison is focused on the fine-tuning stage.
Deepsolo. The performance of the VTS model on ArTVideo is much worse. Moreover, there is a huge gap between the spotting and detection- oolloon-onyyyooo performance of the VTS model, which indicates that the recognition capability is the main bottleneck. We attribute this discrepancy to two key aspects: 1) the model architecture and 2) the training data. First, in terms of model architecture, ITS studies [5, 6] have presented the advantages of employing advanced query formulation for text spotting in DETR frameworks [14, 15]. In contrast, existing Transformer-basee Interest (Rol) components or simply cropping detected text regions for recognition. On the other hand, some studies [16, 17] have indicated that there exists optimization conflict in detection and association during the end-to-end training of MOTR [18]. We hold that TransDETR [12], which further incorporates text recognition into MOTR- based architecture, may also suffer from optimization conffict. Second, regarding the training data, most text instances in current video datasets [13, 10, 19] straight or oriented, and the bounding box labels only quadrilateral, which constrains the data diversity and recognition performance as well. Overall, the limitations in model architecture and data probably lead to the unsatisfactory text spotting performance of the SOTA VTS model. Hence, leveraging model and data knowledge from ITS presents considerable value for VTS.
VTS models still rely on Region of
To achieve this, a straightforward approach is to take an off-the-shed SOTA image text spotter and focus the training efforts on tracking across frames, akin to backing-by-detettin methods. An important question is how to efficiently incorporate Rol-free image text spotter for VTS. In this paper, we propose a simple baseline via lonG and short term Matching, termed GoMatching, which leverages an associates text instances across frames with a strong tracker.
Rol-free image text spotter to identify text from each single frame and
Specifically, we select the state-of-the-ar DeepSolo [5] as the image text spotter and design Long- Short Term Matching based tracker termed LST-Matcher. Initially, to adapt the DeepSolo to video datasets while preserving its inherent knowledge, we freeze Deepsolo and introduce rescoring mechanism. This mechanism entails training an additional lightweight text classifier called rescoring head via efficient tuning, and recalibrating confidence scores for detected instances to mitigate performance degradation caused by the image-video domain gap. The final score for each instance is determined by a fusion operation between the original score provided by the image text spotter and the calibrated score acquired from the rescoring head. The identified text instances are then sent to LST-Matcher for association. LST-Matcher can effectively harnesses both long- and short-term information, making a highly capable tracker. As a result, our bascline significantly surpasses existing SOTA methods by large margin with much lower training costs, as shown in Fig. 1(b).
In summary, the contribution of this paper is threefold. 1) We identify the limitations in current VTS methods and propose a novel and simple baseline, which leverages an off- off-the-shelf image text spotter with a strong customized tracker. 2) We introduce the rescoring mechanism and long- short term matching module to adapt image text spotter to video datasets and enhance the tracker's capabilities. 3) We establish the ArTVideo test set for addressing the absence of curved texts in current video datasets and evaluating the text spotters on videos with arbitrary-shape text. Extensive

---

experiments on public challenging datasets and the established ArTVideoo test set demonstrate the effectiveness of our baseline and its outstanding performance with less training budgets. For example, GoMatching achieves the highest ranking on SUDAR15-vides and DSText. Especially on bilingual dataset BOyText, GoMatching obtains a 45% improvement on MOTA compared to the recorded best performance [11]. On curved text dataset ArT'ideoo, GoMatching also surpasses previous SOTA method [12] by a substantial margin.
2 Related Works
2.1 Multi-Object Tracking
Multi-object tracking methods follow the buuking-by-detectiine (TBD) or tracking-br boakkiin-bb-uurryy propagation (TBQP) pipeline. TBD methods [20-22] employ detectors for localization and then use association algorithms to get object trajectories. Different from extending tracks haame-by-frand GTR [23] proposes to generate entire trajectories at once in Transformer. TBQP paradigm extends query-based object detectors[14 14, 15] to tracking. MOTR [18] detects object locations and serially updates its tracking queries for detecting the same items in the following frames, achieving an end-to-end solution. However, MOTR suffers from optimization conflict between detection and association [16, 17], resulting in inferior detection performance. For the VTS task which additionally involves text recognition, naive way of training all modules end-to-end may also lead optimization conflict. In contrast, we explore inheriting prior knowledge of text spotting from ITS models while focusing on the tracking task.
2.2 Image Text Spotting
Early approaches [24-26] crafted Rol-based modules to bridge text detection and recognition. How- ever, these methods ignored one vital issue, i.e., the synergy problem between the two tasks. To overcome this dilemma, recent Transformer-based methods [27, 3, 28, 6] get rid of the fetters of Rol modules, and chase a better representation for the two tasks. For example, RETR-basee TESTR [4] uses two decoders for each task in parallel. contrast, DeepSolo [S] proposes a unified and explicit query form for the two tasks, without harnessing dual decoders. However, the above methods cannot perform tracking in the video.
2.3 Video Text Spotting
Compared to ITS, existing SOTA VTS methods still rely on Rol for recognition. CoTexx [11] adopts a lightweight text spotter with Masked-Rol. then uses several encoders to fuse features derived from the spotter, and finally feeds them to a tracking head with cosine similarity matching. TransDETR [12] performs detection and tracking under the MOTR paradigm and then uses Rotated-Rol extract features for the subsequent recognizer. They pursue training all modules in an end-to-end manner. In comparison, we explore how efficiently turn a Rol free ITS model into VTS one. We reveal the probability of freezing off-the-shelf ITS part and focusing on tracking, thereby saving training budgets while reaching SOTA performance.
3 Methodology
3.1 Overview
The architecture of GoMatching is presented in Fig. 2. It consists of a frozen image text spotter, rescoring head, and a Long-Short Term Matching module (LST-Matcher). We adopt an outstanding off-the-shelf image text spotter (i.e., DeepSolo) and freeze its parameters, with the aim of introducing strong text spotting capability into VTS while significantly reducing training cost. In DeepSolo, there are p sequences of queries used for final predictions, with each storing comprehensive semantics for a text instance. To alleviate spotting performance degradation caused by the image-video domain gap, we devise a rescoring mechanism, which determines the confidence scores for text instances by considering both the scores from the image text spotter and new trainable rescoring head. Finally, design LST-Matcher to generate instance trajectories by leveraging long-short term information.

---

Toxis
Triecserio fer lscasco
Objeet lDs
ATELMR
Bosndlriies
Seans,
Scares
Predictions
Image Text Spotter
Resoring Head
ST-Marcker
LT- LT-haacker
LSST- Matcher
quertee
Triining oabs
000
Inference oabs
Thenshld
Fitrring
Lass function
Outter
The overall architecture of GoMatching. The frozen image text spotter provides text spotting results for frames. The rescoring mechanism considers both instance scores from the image text spotter and a trainable rescoring head to reduce performance degradation due to the domain gap. Long-short term matching module (LST-Matcher) assigns IDs to text instances based on the queries in long-short term frames. The yellow star sign indicates the final output of GoMatching.
Figure 2:
3.2 Rescoring Mechanism
Owing to the domain gap between image and video datasets, employing a frozen image text spotter for direct prediction may result in relative low recall due to low text confidence, further leading to reduction in end-to-end spotting performance. To ease this issue, we devise a rescoring mechanism via a lightweight rescoring head and simple score fusion operation. Specifically, the rescoring head is designed to recompute the score for each query from the decoder in the image text spotter. It consists of simple linear layer and is initialized with the parameters of the image text spotter's classification head. The score fusion operation then decides the final scores by considering both the scores from the image text spotter and the rescoring head. Let C = { .... o be set of original scores produced by image text spotter in frame t. C = " ..., cf, is a set of recomputed scores obtained from the rescoring head, We obtain the maximum value for each query as the final score, denoted C* {ch max(c', , (f,),.., Sf max(c (co, o,,)). )]. With final scores, the queries in frames are filtered by threshold before being sent to LST-Matcher for association.
3.3 Long-Short Term Matching Module
+-1
Long-short term matching module (LST- Matcher) consists of two sub-modules: the Short Term Matching module (ST-Matcher) and the Long Term Matching module (LT-Matcher), which own the same structure. ST-Matcher is steered to match simple instances between adja- cent frames into trajectories, while LT- Matcher responsible for using long term information to address the unmatched instances due to severe occlusions or strong appearance changes. Each of them contains one-layer Transformer coder and a one-layer Transformer decoder [23].
Et
ST-Matcher
exbadding
LT-Matcher
Fash
Figure 3: The inference pipeline of LST-Matcher, which is a two-stage association process: (1) ST- Matcher associates the instances with trajectories previous frames denoted by blue lines. (2) LT-Matcher associates the remaining unmatched instances by utilizing other trajectories in history frames as denoted by red lines.
We use simple multi-layer perceptron (MLP)
to map the filtered text instance queries into em- beddings the input, getting rid of using RoI features as in most existing MOT methods. In the encoder, historical embeddings are enhanced by self atteettnnoo. The decoder takes embeddings the current frame as query and enhanced his- torical embeddings as key for cross-attention, and computes the association score matrix. The current instances are then linked to the existing trajectories composed of historical embeddings or generate new trajectories according to the association score matrix.

---

To be specific, supposing a given clip including T frames and N; text instances in frame t after threshold filtering. Q' {qi.... LN,1 is the set of text instance queries in frame t. Initially, we use a two-layer MLP to map these frozen queries into embeddings E = {ei, e.,,). The set of embeddings in all frames denoted as E = E² J... U E" Let the universal set of embeddings in adjacent frames of the input batch be denoted F F U U...U EST and F.S. F Based on the predictions of image text spotter, we obtain their corresponding bounding boxes Be {bf.... ..i....66 J., Let (TI..., ] be the of ground-truth (GT) trajectories of all instances in the clip, where Tę = (Tz,..., ...; T describes a tube of instance locations oE E IR U {0) through time. o 0 means the absence of instance k in frame t. Let á be the matched instance index for Ti according to the following equation:
6. T 0 or max; (loU(b) +B)) < 0.5 argmax, argmax, (loU(,) +(),, otherwise
at -
ST-Matcher calculates short-term association score o/ ES) Ns, for i-th instances in frame t, where eaa E long-term srajeciory-specifie association score (ex,, F.4) R.V for i-th instances in frame t, where ex {eat 23 -., J Σi= N.. Specifically, when v/(e E
I is trajectory query and Ns. =N+ + N1-1- LT-Matcher calculates
and (e E*)
it means no association at time t. Then, ST-Matcher and LT-Matcher can predict distributions of short-term and long-term associations for all instance in frame & which can be written as:
exp(v](e's ES:))
Ps. (e'g ESi)
(2)
Σ _JEiDI....NN. exp exp(u) (5 ES)))
exp(u( E²))
Pt (ek, E²) (je(0.1.....)) exp( (uf (ek, E4))
To ensure sufficient training of ST-Matcher and LT-Matcher, embeddings set E and E are fed into ST-Matcher and LT-Matcher during training, respectively.
During inference, we engage memory bank to store the instance trajectories from H history frames for long term association. All filtered instances in each frame are further processed by con-maximum- suppression (NMS) before being fed into LST-Matcher for association. Unlike the training phase, where ST-Matcher and LT-Matcheder are independent of each other, LST-Matcher comprises two-stage associating procedure as described in Fig. Concretely, ST-Matcher first matches the embedding E' in the current frame / with the trajectories 7'-1 in the previous frame -- 1. Then, Hf LT-Matcher employs other trajectories Toth in the memory bank to associate the unmatched ones E with low association score in ST-Matcher caused by the heavy occlusion or strong appearance changes. the association score with any trajectory calculated in ST-Matcher or LT-Matcher higher than threshold 0, the instance is linked to the trajectory with the highest score. Otherwise, this instance is used to initiate a new trajectory. Finally, we combine the trajectories т, and T predicted by ST-Matcher and LT-Matcher to obtain new trajectories 7N for tracking in the next frame.
3.4 Optimization
Rescoring Loss. To train the rescoring head, we following DETR [14] and use Hungarian algo- rithm [29] to find a bipartite matching â between the prediction set Y and the ground truth set with minimum matching cost C:
arg min Σ C(Y., Vo(o))
(4)
where N is the number of ground truth instances per frame. The cost can be defined as:
C(Y,, Yo(g) XcCols(6 Ao
Σ bb bo(e)
(5)
where da and A serve as the hyper-parameters to balance different tasks. (c;) and (y are the probability for ground truth class c, and the predicition of bounding box respectively, and 6,
5

---

represents the ground truth bounding box. Lets is the focal loss [30]. Specifically, the focal loss for training the rescoring head can be formulated as:
Cres -1 (axee (aarr:a0"0eDuuuuaooeeoooaaooaa/e 1og
log(1- (c))),
(6)
where a and 7 are the hyper-parameters of focal loss.
Long-Short Association Loss. In ST-Matcher, we only consider each trajectory in the universal set of adjacent frames, while in LT-Matcher we consider each trajectory in all long term frames. For each trajectory, optimize the log- rg-likelihood of its assignments 0g following GTR [23]:
L ass E [th)
log Ps. (á) ea E
2=
C Ci (k =- -- log Pt, (á E.d E
(8)
where w E {1,..., T D}.
In ST-Matcher and LT-Matcher, empty trajectories would be generated for these unassociated queries, and their optimization goals can be defined as:
L. ba =- Σ
log Ps, (a' = 01e/), ES:),
(9)
j: =j
T
C (E4)
Σ
Σ log Pt, (a' =
(a'=a 01E E²).
(10)
w=1
j:
Finally, can get the long-short association loss follows:
Lasso = Las + Li_bi bg + E(Ss + La assss)
(11)
P2
Overall Loss. Combined with the rescore loss Cres in Eq. (6) and the long-short association loss Casso in Eq. (11), the final training loss can be defined as:
L= dreshres + AassoLasso: where the hyper-parameters A and Aasso are the weights of Cres and Lassor respectively.
(12)
Experiments
4.1 Datasets and Evaluation Metrics
ICDARIS-video [13] is a word-level video text reading benchmark annotated with quadrilateral bounding boxes, comprising training set of 25 videos and test set of 24 videos. It focuses wild scenarios, such as driving on the road, exploring shopping streets, walking in a supermarket, etc.
BOVText [10] is large-scale, bilingual, and world benchmark for video text spotting, encom- passing English and Chinese. The dataset is meticulously collected from YouTube and KuaiShou with different scenarios. The text box annotations are represented quadrilaterals at the textline level.
DSText [19] is newly proposed dataset, and focuses dense and small text reading challenges in the video. This dataset provides 50 training videos and 50 test videos. Compared with the previous datasets, DSText mainly includes the following three new challenges: dense video texts, nigh-proportionel small texts, and various new scenarios, e.g., 'Game', 'Sports', etc. Similar to ICDAR15-video, DSText adopts word-level annotations, which are labeled with quadrilaterals.
ArTVideo novel word-level test set established in this work to evaluate the performance of arbitrary-shaped video text, which is absent in the VTS community. It contains 20 videos with about 30% curved text instances. Straight text is annotated with quadrilaterals, while curved text is annotated with polygons. More details are provided in the Appendix A.
Evaluation Metrics. To evaluate performance, we adopt three evaluation metrics commonly used in ICDAR 100AR15-vii vivio competition and DSText competition, including MOTA [3]] MOTP, and IDFI [32].

---

4.2 Implementation Details
In all experiments, we only use a single NVIDIA GeForce RTX 3090 (24G) GPU to train and evaluate GoMatching. As for the image text spotter in GoMatching, we apply the officially released DeepSolo [5]. During fine-tuning GoMatching on downstream video datasets, we only update the rescoring head and LST-Matcher, while keeping DeepSolo frozen. More inference settings can be seen in the Appendix B.
Training Setting. The text spotting part of GoMatching is initialized with off offfthheshhef DeepSolo weights and kept frozen in all experiments. We optimize other modules on video datasets. We follow EfficientDet [33] adopt the scale-and-crop augmentation strategy with a resolution of 1280. The batch size T is 6. All frames in a batch are from the same video. Text instances with fusion scores higher than 0.3 are fed into the LST-Matchhr during training. AdamW W [34] is used as the optimizer. We adopt the warmup cosine annealing learning rate strategy with the initial learning rate being set to 0.00005, The loss weights Ares and Aasoo are set 1.0 and 0.5, respectively. For focal loss, a is 0.25 and 7 is 2.0 as in [14, 5]. The model is trained for 30k iterations on all downstream video datasets.
4.3 Comparison with State-of-the-art Methods
Results on ICDARIS-video. To evaluate the effectiveness of GoMatching on oriented video text, we conduct comparison with the state-of-the-arr methods on ICDAR15-vides presented in Table la. As can be seen, GoMatching ranks first in all metrics on the CEDAR15-video leaderboard. By effectively combining robust image text spotter with strong tracker, GoMatching improves the best performance by 5.08% MOTA, 0.75% MOTP, and 3. 3.16% IDF1, respectively. Furthermore, owing to the substantial enhancement in recognition and tracking capabilities (details can be found in Sec. 4.4 and Appendix G), GoMatching outperforms the current SOTA single-mode model method TransDETR by 11.08% MOTA, 3.92% MOTP, and 7.31% % IDFI, respectively.
Results on BOVText. Except for the English word recognition scenario, GoMatching can readily adapt other video text recognition scenarios, such Chinese text line recognition. For BOVText, which focuses on English and Chinese textline recognition, we employ the DeepSolo trained on bilingual textline datasets and then fine-tune GoMatching on BOVText. The results are presented in Table lb. It is evident that GoMatching achieves new record on the BOVText dataset and surpasses previous methods significantly. GoMatching exhibits superior performance over the previous SOTA method CoTTxx [11], with improvements of 41.5% on MOTA, 6.9% on MOTP, and 14.3% on IDF1. Such exceptional performance of GoMatching on BOVText suggests its proficiency spotting both Chinese and English text in videos. Moreover, it can be easily extended to other languages by adapting the image spotter.
Results on DSText. We further conduct experiments on DSText with dense and small video text scenarios. Results are presented in Table 1c. It is worth noting that most of the previous methods on
the DSText leaderboard used an ensemble of multiple models and large public datasets to enhance their performance [19]. For example, TencenrOCR integrates the detection results of DBNet [35] and Cascade MaskRCNN [36] built with multiple backbone architectures, combines the Parseq [37] text recognizer, and further improves the end-to-end tracking with ByteTTrack [22]. DA adopts Mask R-CNN [38] and DBNet to detect text, then uses BotSORT [21] to replace the tracker in VideoTextSCM [39] and employs the Parseq model for recognition. As a single model with a frozen image text spotter, GoMatching also shows competitive performance compared to other ensembling methods on the leaderboard. GoMatching ranks first (22.83%) on MOTA, second (80.43%) on MOTP, and third (46.09%) on IDF1. Moreover, compared to the SOTA single-model method, GoMatching achieves substantial improvements of 45.46% and 19.66% on MOTA and IDF1, respectively.
Results on ArTVideo. text spotting capabilities for arbitrary-shaped text. For a fair comparison, both TransDETR and GoMatching are trained on ICDAR15-viden Unlike ICDAR15-video and DSText which only have straight text, ArTVideo has substantial number of curved text, so we report results under four settings: tracking results on both straight and curved text, spotting results on both straight and curved text, tracking results on curved text only, and spotting results on curved text only. As shown in Table 1d, GoMatching outperforms TransDETR under all settings. Especially when involving an additional recognition task (end-to-end spotting) or only considering curved text, the performance advantages of GoMatching are more significant. This further confirms that the previous SOTA
We test TransDETR and GoMatching on ArTVideo to compare the zero-shot
7

---

Table 1: Comparison results with SOTA methods on four distinct datasets. 't' denotes that the results are collected from the official competition website. *#: we use the officially released model for evaluation. 'M-ME' indicates whether multi-modee ensembling is used. "Y' and 'N' stand for yes and no. The best and second-best results are marked in bold and underlined, respectively.
(a) Results ICDAR15-viden
(h) Results on BOVText.
Method
MOTP
(*)
Method
MOLA
MOIP (T)
IDFI (T)
HIK_ OCR [9]
52.98
74.88
61.85
EAST CRNN [T]]
-79.3
76.3
6.8
CoText [1]]
58.94
74.53
71.66
PSENet + CRNN [10]
-17.0
79.2
31.3
TransDETR [12]
60.96
74.61
72.80
DB CRNN [10]
81.3
labt_Jabt
63.76
77.78
71.08
VTSptter [10]
-1.4
82.0
43.6
GOCR Ofline!
63.05
74.31
76.95
CoText [1]]
80.3
48.3
CoTest CoText(Kuisson Kunishou MMUJy
66.96
76.55
74.24
GoMaiching (ours)
52.9
87.2
62.6
GoMatching (size:800) (ours)
68.51
77.52
76.59
GoMatching 1000) (ours)
72.04
78.53
80.11
(d)
Results on ArTVideo.
GoMatching (size: 1440) (ours)
70.52
78.25
78.70
Method
MOTA MOTP(T)
IDFT
(c) Results on DSText.
ArTVidee Tracking
TransDETR [12]
54.2
67.9
70.4
Method
M-ME MOTA (1)
MOTP (?)
IDFT (7)
GoMatching (ours)
81.3
HRNN
Y
28.58
80.36
26.20
A/T Video Enn-to-EEnd Spotting
SCUT-MMOCR-KK
-27.47
76.59
45.61
TransDETR [12]
2.8
69.7
49.3
TextTrocke
Y
-25.09
74.95
26.38
GoMatching (ours)
82.9
78.5
abcmoot?
Y
5.54
74.61
24.25
ArT Video-Curved Tracking
DAT
Y
10.51
78,97
53.45
TransDETR [12]
4.4
60.5
50.2
TencentCCF
Y
22.44
80.82
$6.45
GoMatching (ours)
59.5
76.3
73.5
TEaR [12]
-22.33
79.75
26.43
Ar ArTViden-Cnun Curved Fnd to- Fnd Spotting
GoMatching (ours)
N
22,83
80.43
46.09
TransDETR [12]
-66.7
6T.9
26.9
GoMatching (ours)
56.8
78.0
73,9
methods have unsatisfactory recognition capabilities and limited adaptability to complex scenarios. Furthermore, as shown in Fig. (b), GoMatching achieves excellent performance while significantly reducing the training budget.
Some visual results are provided in Fig. 4. It shows that GoMatching performs well on straight and curved text, and even more complex scene text. More visual results (including some failure cases) and analysis are provided in the Appendix G.
4.4 Ablation Studies
We first conduct comprehensive ablation studies on ICDARI5-video to verify the effectiveness of each component. The experimental results are shown in Table 2. The impact of frame length on long-term association during inference is then studied, and the results are shown in Appendix C.
Effectiveness Utilizing Queries. Comparing the first two rows in Table 2, we can find that using queries from the decoder of image text spotter is more beneficial for tracking than Rol features. By leveraging the unified queries from frozen DeepSolo, 0.98% and 1.05% improvements on MOTA and IDFI are achieved. This is because queries integrate more text instance information, i.e., unifying multi-scale features, text semantics, and position information, which has been proven effective in DeepSolo. Although position information is essential for tracking, it is ignored in Rol features.
Effectiveness of Rescoring Mechanism. To verify the effectiveness the rescoring mechanism, we test three different scoring mechanisms: the original score from DeepSolo, the score recomputed by the rescoring head, and the fusion score from the rescoring mechanism. As shown in row and row of Table 2, the rescoring head can alleviate the performance degradation caused by the domain gap between ICDAR 1 5-image and ICDARIS 5-vvddoo and achieve gains of 1.25% and 0.97% on MOTA and IDFI, respectively. Moreover, as shown in row 4, can observe that combining the knowledge of rescoring head learned from the new dataset with the prior knowledge of DeepSolo can further improve MOTA and IDFI by 0.33% and 0.32%, respectively. Appendix F contains more results.
Effectiveness of LST-Matcher. In this part, we conduct three experiments to prove the effectiveness of the LST-Matcher. As shown in row 4 of Table 2, we only use LT-Matcher to associate high-score text instances in the current frame with trajectories in the tracking memory bank. In row 5, we only use ST-Matcher to associate high-score text instances in the current frame with trajectories of the previous frame. In addition, as shown in row 6, we employ both LT-Matcher and ST-Matcher to test L.T Matcher. We can easily observe that compared to LT-Matcher, LST- improves MOTA
8

---

ROGHLE
ROCHEUE
REcaa
Figure 4: Visual results of video text spotting. Images from top bottom are the results ICDAR15-vider BOVText, DSText, and ArTVideo, respectively. Text instances belonging to the same trajectory are assigned the same color.
Table 2: Impact of difference components in the proposed GoMatching. 'Query' indicates that LST-Matcher employs the queries of high-score text instances for association, otherwise Rol features. Column 'Scoring' indicates the employed scoring mechanism, in which 'O" means using the original scores from DeepSolo, 'R' means using the scores recomputed by the rescoring head, and 'F' means
using the fusion scores obtained from the rescoring mechanism.
Query
Scoring
LT-Marcher
ST-Matcher
MOTA
MOTP (?)
IDFT (f)
66.20
78.52
75.07
67.22
78.54
76.12
68.47
78.29
77.09
68.30
78.24
77.41
69.40
78.34
73.60
70.52
78.25
78.70
and IDFI by 1.72% and 1.29% respectively, while compared to ST-Matcher, the improvement on
MOTA and IDFI are 1.12% and 5.1%, respectively. In Fig. 5, we also demonstrate that using LST- Matcher can effectively mitigate the issue of ID switches caused by the strong appearance changes due to motion blur. These results validate that combining short-term and long-term information leads to more robust tracking outcomes, thereby enhancing the performance of video text spotting.
Different training strategies. To investigate the impacts of different training strategies GoMatch- ing, we establish three distinct settings on the ICDARI5-video dataset, as shown in Table 3. 1) We only fine-tune the tracker while keeping the image text spotter frozen (the first row of Table 3). 2) We first fine-tune the image text spotter on images extracted from ICDAR 5-video and then further fine-tune the tracker with the image text spotter fixed (the second row of Table 3). 3) We jointly train the spotter's decoder and tracker of GoMatching while trying different learning rates for the decoder (the last three rows of Table 3). As shown in the first two rows of Table 3, fine-tuning the image spotter on the downstream video dataset results in a performance decline compared to the default setting. This is due two data-related factors: 1) minor variations text content between frames in the same video lead to insufficient data diversity, causing the image text spotter to overfit more easily; and 2) image blurring from camera motion reduces the quality of data available for training the image text spotter. When the image text spotter and tracker are trained simultaneously (the last three rows), the model's performance significantly decreases. Even with the decoder's learning rate close to zero (i.e., 0.00/), there is still 1.13% drop in IDF1. As the decoder's 's learning rate increases, the performance decline becomes more pronounced. This indicates that naive joint optimization of text spotting and tracking is challenging, likely due to conflicts between the two tasks. In future work.

---

ID Switches
Figure 5: Visualization results ST-Matcher and LST-Matcher. The first row shows the failure case that suffers from the ID switches issue when using only ST-Matcher, caused by missed detection and erroneous matching. The second row shows that LST-Matcher effectively mitigates this issue via both long and short term matching. Text instances in the same color represent the same IDs.
Table 3: Results of GoMatching under various training settings on the ICDARI5-vides dataset. 'Only Image Spotter' and 'Only Tracker' refer to fine-tuning either the image spotter tracker with another module fixed. Band-to-End' denotes that training the image spotter and tracker in an end-to-end manner. '0.001', '0.01" and '0.1" correspond to the ratios of the learning rate employed by the decoder of the image text spotter relative to the base learning rate. Due to constraints training resources, we only optimize the parameters of the decoder component for the image text spotter.
Index
Training Setting
MOTA (f)
MOTP (t)
IDFI (1)
OOly Tracker'
72.04
78.53
80.11
2
First 'Only Image Spotter', Then 'Only Tracker'
70.82
78.09
79.64
3
'End-to-EE
'End-to-Endd, End", Image Spotter's Decoder ('0.001')
71.48
79.14
78.98
'End End', Image Spotter's Decoder ('0.01")
70.15
78.17
77.67
'End-to-Ead', Image Spotter's Decoder ('0.1")
68.03
75.46
77.16
it is worth trying to establish larger, more diverse video text spotting datasets and to explore more effective multi-task optimization strategies.
Conclusion
In this paper, we propose simple yet strong baseline, termed Go.Mtthhngg, for video spotting. GoMatching harnesses the talent of an off-the-shelf query-based image text spotter and only needs to tune lightweight tracker, effectively addressing the limitations of previous SOTA methods in recognition. Specifically, we design the rescoring mechanism and LST-Matcher to adapt the image text spotter to unseen video datasets while empowering GoMatching with excellent tracking capability. Moreover, we establish novel test set Arl'Video for the evaluation of video text spotting models on whhtee text, filling the gap in this area. Experiments on public benchmarks and ArTVideo demonstrate the superiority of our GoMatching in terms of both spotting accuracy and training cost.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Key Research and Development Program of China under Grant 2023 in part by the National Natural Science Foundation China under Grant U23B2048, 62076186 and 62225113, 13, in part by the Innovative Research Group Project of Hubei Province under Grant 20241F and part by the Science and Technology Major Project of Hubei Province under Grant 2024BAB046. Dr Tao's research is partially supported by NTU RSR and Start Up Grants. The numerical caleulations in this paper have been done on the supercomputing system in the Supercomputing Center of Wuhan University.
10

---

References
[1]
Jianfeng Dong, Xirong Li. Chaoxi Xu, Xun Yang, Gang Yang. Xun Wang, and Meng Wang. Dual encoding for video retrieval by text. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(8):4065-4088 2021.
[2] Chongsheng Zhang, Yuefeng Tao, Kai Du, Weiping Ding, Bin Wang, Ji Liu, and Wei Wang. Character-level street view text spotting based on deep multisegmennatioon network for smarter autonomous driving. IEEE Transactions on Artificial Intelligence, 3(2):297-308, 2021.
[3] Yuliang Liu, Jiaxin Zhang, Dezhi Peng, Mingxin Huang, Xinyu Wang, Jingqun Tang, Can Huang, Dahua Lin, Chunhua Shen, Xiang Bai, et al. Spts v2: single-point scene spotting. arXiv preprint 23Xv:2301.00633. 2023.
[4] Xiang Zhang, Yongwen Su, Subarna Tripathi, and Zhuowen Tu. Text spotting transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 9519-9528, 2022.
[5] Maoyuan Ye, Jing Zhang, Shanshan Zhao, Juhua Liu, Tongliang Liu, Bo Du, and Dacheng Tao. Deepsolo: Let transformer decoder with explicit points solo for text spotting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 19348-19357, 2023.
[6] Mingxin Huang, Jiaxin Zhang, Dezhi Peng, Hao Lu, Can Huang, Yuliang Liu, Xiang Bai, and Lianwen Jin. Estextspotter: Towards better scene text spotting with explicit synergy in transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19495-19505, 2023.
[7] Xiaobing Wang, Yingying Jiang, Shuli Yang. Xiangyu Zhu, Wei Li, Pei Fu, Hua Wang, and Zhenbo Luo. End-to-end scene text recognition in videos based on multi frame tracking. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1,
pages 1255-1260. IEEE, 2017.
[8] Zhanzhan Cheng, Jing Lu, Yi Niu, Shiliang Pu, Fei Wu, and Shuigeng Zhou. You only recognize once: Towards fast video text spotting. In Proceedings of the 27th ACM International Conference on Multimedia, pages 855-863, 2019.
[9] Zhanzhan Cheng, Jing Lu, Baorui Zou, Liang Qiao, Yunlu Xu, Shiliang Pu, Yi Niu, Fei Wu, and Shuigeng Zhou. Free: A fast and robust end-to-end video text spotter. IEEE Transactions on Image Processing, 30:822-837, 2020.
[10] Weijia Wu, Yuanqiang Cai, Debing Zhang, Sibo Wang, Zhuang Li, Jiahong Li, Yejun Tang, and Hong Zhou. bilingual, openworld video text dataset and end-to-end video text spotter with transformer. arXiv preprint arXiv:2 /2. 02.8888, 2021.
[1I] Wejia Wu, Zhuang Li, Jiahong Li, Chunhua Shen, Hong Zhou, Size Li, Zhongyuan Wang, and Ping Luo. Real-time end-to-end video text spotter with contrastive representation learning. arXiv preprini arXiv: 2207.2220008444 07 2022.
[12] Weijia Wu, Yuanqiang Cai, Chunhua Shen, Debing Zhang, Ying Fu, Hong Zhou, and Ping Luo. End-to-end video text spotting with transformer. International Journal of Computer Vision, 132 (9):4019-4035, 2024.
[13] Dimosthenis Karatzas, Lluis Gomez-Bigorde Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In 2015 13th international conference on document analysis and recognition (ICDAR), pages 1156 1160. IEEE, 2015.
[14] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[15] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.

---

[16] Yuang Zhang, Tiancai Wang, and Xiangyu Zhang. Motrv2: Bootstrapping end-to-end multi- object tracking by pretrained object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22056-22065. 2023.
[17] En Yu, Tiancai Wang, Zhuoling Li, Yuang Zhang, Xiangyu Zhang, and Wenbing Tao. Motrv3: Release-fetch supervision for end-to-end multi-object tracking. arXiv preprint arXiv:2305. 14298, 2023.
[18] Fangao Zeng, Bin Dong, Yuang Zhang. Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end muleep tracking with transformer. In European Conference on Computer Vision, pages 659-675. Springer, 2022.
[19] Weijia Wu, Yuzhong Zhao, Zhuang Li, Jiahong Li, Mike Zheng Shou, Umapada Pal, Dimosthe- nis Karatzas, and Xiang Bai. Icdar 2023 video text reading competition for dense and small text. arXiv preprint arXiv:2304.04376 2023.
[20] Zhongdao Wang. Liang Zheng, Yixuan Liu, Yali Li, and Shengjin Wang. Towards multi-object tracking. In European Conference Computer Vision, pages 107-122. Springer, 2020.
[21] Nir Aharon, Roy Orfaig, and Ben-Zion Bobrovsky. Bot-sort: Robust associations multi- pedestrian tracking. arXiv preprint arXiv:2206. 2e222226..44770 14651, 2022.
[22] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In European Conference on Computer Vision, pages 1-21. Springer, 2022.
[23] Xingyi Zhou, Tianwei Yin, Vladlen Koltun, and Philipp Krähenbühl. Global tracking transform- In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9771-8780. 2022.
[24] Minghui Liao, Guan Pang, Jing Huang, Tal Hassner, and Xiang Bai. Mask textspotter v3: Segmentation proposal network for robust scene text spotting. In Computer Vision-ECC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 706-722. Springer, 2020.
[25] Wenhai Wang, Enze Xie, Xiang Li, Xuebo Liu, Ding Liang, Zhibo Yang. Tong Lu, and Chunhua Shen. Pan++: Towards efficient and accurate end-to-end spotting of arbitrarily-shaped text. IEEE Transactions on Pattern Analysis and Machine Intelligence, (4(9):539-53366 2021.
[26] Yuliang Liu, Chunhua Shen, Lianwen Jin, Tong He, Peng Chen, Chongyu Liu, and Hao Chen. Abenet v2: Adaptive bezier-curve network for real-time end-to-end text spotting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1 1):8048-80064 2021.
[27] Yair Kittenplon, Inbal Lavi, Sharon Fogel, Yarin Bar, R Manmatha, and Pietro Perona. Towards weakly-supervised text spotting using multi-task transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4604 4613, 2022.
[28] Maoyuan Ye, Jing Zhang, Shanshan Zhao, Juhua Liu, Tongliang Liu, Bo Du, and Dacheng Tao. Deepsolo++: Let transformer decoder with explicit points solo for multilingual text spotting. arXiv preprint arXiv:2305. 19957, 2023.
[29] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[30] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[31] Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance: The clear mot metrics. EURASIP Journal on Image and Video Processing, 2008:1-10, 2008.
[32] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance measures and a data set for multi-target, multi-camera tracking. In European conference on computer vision, pages 17-35. Springer, 2016.
12

---

[33] Mingxing Tan, Ruoming Pang, and Quoe V Le. Efficientdet: Scalable and efficient object detec- tion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 10781-10790, 2020.
[34] Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,2000 2017.
[35] Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai, Real-time scene text detection with differentiable binarization. In Proceedings of the AAAl conference artificial intelligence, pages 11474-1 1481, 2020.
[36] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance segmentation. IEEE transactions on pattern analysis and machine intelligence, pages 1483- 1498, 2019.
[37] Darwin Bautista and Rowel Atienza. Scene text recognition with permuted autoregressive sequence models. In European Conference on Computer Vision, pages 178-196. Springer, 2022.
[38] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn. In Proceedings of the IEEE international conference computer vision, pages 2961-2969, 2017.
[39] Yuzhe Gao, Xing Li, Jiajian Zhang, Yu Zhou, Dian Jin, Jing Wang, Shenggao Zhu, and Xiang Bai. Video text tracking with spatio-temporal complementary model. IEEE Transactions on Image Processing, pages 9321-9331, 2021.
[40] Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang. Curved scene text detection via transverse and longitudinal sequence connection. Pattern Recognition, 90:
337-345, 2019.
[41] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Big- orda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. Icdar 2013 robust reading competition. In 2013 12th international conference document analysis and recognition, pages 1484-1492 IEEE, 2013.
13

---

A More Details of ArTVideo
Due to the scarcity of curved text instances within existing video text spotting datasets, it is infeasible to evaluate the performance of video text spotting models on curved text. To fill this gap, collected a test set named ArTVideo containing 20 video clips with a total of 884 frames, in which 18 videos were collected from YouTube and 2 videos from the BOVText test ArTVideo contains 6,526 text instances, including 4,632 straight text instances and 1,894 curved text instances, i.e., curved text accounts for about 30%. As shown in Fig. 6, we provide high-quality word-level annotations for both straight and curved text in two different annotated ways. The straight text is labeled with quadrilaterals, while for curved text, we follow the CTWI 1500 [40] and adopt a polygon with 14 points to annotate the text contour. More statistics about Video are shown in Fig. 7.
BINGJENEE WHANE CLAR HEVE WAREET
1 will
BBRAND RETAIL MODEL
BRLTL-BRADD RETAIL MODEL
BRAND RETAIL FODEL
Figure 6: Visual examples from our ArTVideo. The straight and curved text are labeled with quadrilaterals and polygons, respectively. The same background color in different frames (columns) denotes the same instance.
ArTVa
ArTVi
Porestage (74)
Percentage (2))
Instnce Claants
Frente
Leatlen per Instance
(a)
CTORR
M
Ter
NEXT
0104
LIVITED Electrolit
2210K
SHEFFIELD ELD
BEACH
CONTRACTORE
530kcoL Kop!
1ST
Floar
BRAMALE
TACO
HARD MI Kiaaa
KINEMASTER
MEDNESON SONY
FALORA
FOOTEM
FOOTBALL
Weppoces
BICYELE
1862
INDLA
25
SEACREST
MODEL
FOR
CARUARY KBAC putu
15
clubak
1889
1855
MASTEROT
STAGES NHI HALLAM
CREAM
553kj
FOO
JEREYS
BELL
UNTTED
ON KD Made
URS ANL
pICLL LECAAN
IT B
STINGLE
THE
USA
BURLING
CONS CONS-LiD LiD CRICKEY
GAME
The L998
R RETAT
BETWEEN
1BEKCOLMUL
(c)
Figure 7: Statisties of ArTVideo. (a) and (b) show the distribution of text instance numbers in each frame and the distribution of the text length of each instance, respectively. (c) shows the word cloud of text annotations in ArTVideo.
14

---

Table 4: Ablation studies the number of frames (T) for long-term association in LT-Matcher, and the max number of history frames in tracking mem- ory bank is H = T - 1). Experiments are con- ducted ICDARIS-vide and the best results marked in bold.
Table 5: Results of different score fusion strategies on ICDARS-video. 'Mean', 'Geo- mean', and 'Maximum' denote the arithmetic mean, geometric mean, and the maximum score fusion strategies, respectively. The best results
are highlighted in bold.
Number 7
MOTA (t)
MOTP (t)
IDFI (1)
T 32
70.13
78.07
78.24
Strategy MOTA (1)
(TTP
IDFI (f)
T' 16
70.33
78.25
78.60
Mean
70.46
78.38
78.29
T
70.44
78.25
78.70
Geo-mean
70.29
78.39
78.26
T 6
70.52
78.25
78.70
Maximum
70.52
78.25
78.70
T 4 70.51
78.27
78.16
Table 6: Results of using different image sizes on ICDARI5-video. 'Size' means the size of the shorter side of the input image during inference.
Table 7: Comparison between TransDETR and GoMatching. "T-Para." and "A-Para." de- note the number of all parameters and the train- able parameters in each model, respectively,
The best results are highlighted in bold.
Method
(TTA(T)
MOTP (1)
IDF1 (T) FPS
Method #T-Para. (M)
#A #A-Para. (M)
DEnT(DETRSSize 800)
60.96
74.61
12.80
12.69
TransDETR 39.35
39.58
GoMatching Size:
68.51
14.41
(oMaiching(Size: 1000)
72.04
78.53
80.11
10.60
GoMatching
32.79
75.38
B More Details of Inference Settings
Since there is no training set in ArTVideo, we directly use the model trained on ICDARI5-video to evaluate the generalization ability of GoMatching arbitrary-shaped text. The association score threshold is set to 0.2. For ICDARIS CCDAR15-video, we set the shorter size of the input image to 800, 1000, and 1440, with 800 aligned with the setting in TransDETR and 1440 aligned with the setting in DeepSolo. As for BOVText, DSText, and ArT'Vidoo, the shorter sizes are set to 1000, 1280, and 1440, respectively. All the ablation studies are conducted on the setting of 1440.
C Impact of the Frame Number in LT-Matcher
In Table 4, we further study and analyze the impact of the number of frames for long-term association I.T-Matcher during inference. For the text spotting task, since a single frame may have large number of text instances, sometimes reaching hundreds, excessive historical frame information would weaken the discrimination text instance features, resulting in erroncous matching results. Therefore, conduct paramete search and find that the optimal frame number is 6.
D Comparison of Different Score Fusion Strategies
To investigate the impact of the score fusion strategy in the rescoring mechanism, we further evaluate two other strategies: (1) the arithmetic mean fusion strategy and (2) the geometric mean score fusion strategy, denoted as mean and geo-mean, respectively. The arithmetic mean score fusion strategy takes the average of the scores from the image text spotter and the rescoring head as the final score for each query, while the geo-mean score fusion strategy uses the geometric mean. These two strategies be formulated as:
Cmnann = (Co + c22/2,
(13)
Cgeo-mean /Co * Cr,
(14)
where Crean and Cyeo-meat mean denote the final score of the two strategies, respectively. c, and c, the scores from the image text spotter and the rescoring head, respectively.
From Table 5, we can see that employing the maximum score fusion strategy achieves the best performance on MOTA and IDFI among all the strategies. As for the other two strategies, extremely low confidence scores from the image text spotter may result in low final scores, probably leading to missed detections. Therefore, we adopt the maximum score fusion strategy in the rescoring mechanism of GoMatching by default.
15

---

With Rescoring Mechanism
Without Rescoring Mechanism
FRANKEURT FRAN
CER
RESIIE
serv Metsarre
servis Motes
servi
vervi Iaareeate
servi UNNEEOSt
Figure 8: Visual comparison of GoMatching with and without the rescoring mechanism. The values in parentheses indicate the confidence scores of the detected text. "-² points to the filtered texts due to low confidence without using rescoring. '-*' points to the whose confidence score has been improved by the rescoring mechanism. The rescoring mechanism increases the confidence scores of small texts and blurry texts, preventing them from being filtered out by the threshold and thereby cultivating a better tracking candidate pool.
Table 8: Video text detection performance ICDAR2013-vides [41]. The best results are high- lighted in bold.
Table 9: Comparison results of detection AP the ICDAR13-video between with and without the rescoring mechanism.
Method
Precision
Recall (?)
F-measure (T)
79.7
73.6
Method
AP
APs
APM
AP:
TEIRDETR
80.6
70.2
75.0
weo rescoring
26.2
11.6
40.1
49.8
GoMatching w/o rescoring
92.4
w/ rescoring
29.3
(43.1)
15.5 (+3.9)
42.7 (42.6)
(-2.1)
GoMatching
89.5
74.8
81.5
E More Comparisons between TransDETR and GoMatching
In Table 6, we provide the results of using three different image sizes in GoMatching during inference on CCDAR15-videss In the first row of Table 6, the shorter side of the input image is set to 800, which is the same as the default setting of TransDETR. It is evident from Table 6 that GoMatching significantly outperforms TransDETR in settings. Meanwhile, it also shows that GoMatching outperforms TransDETR under its default setting in terms of both inference speed and spotting
16

---

LAPLASS BEST
Garchign
Figure More visualization results of TransDETR and GoMatching. Failure cases of text detection and recognition are highlighted with ellipses and rectangles, respectively.
accuracy. With the increase in image size (e.g., 1000), GoMatching offers better spotting performance the cost of decreased inference speed.
Moreover, we compare the number of parameters and the trainable parameters of GoMatching and TransDETR, as shown in Table 7. It is noteworthy that GoMatching requires much smaller training budget than TransDETR owing to its simpler architecture design, making a simple but strong baseline for video text spotting.
F More Results for the Rescoring Mechanism
In Table 8, we provide the video text detection results of GoMatching on ICDARI --vidoo [41] and compare them with Free and TransDETR. As shown in the table, without the rescoring mechanism. GoMatching relies on the original results from DeepSolo, resulting in a 4.5% decrease on Recall and only a 1.8% improvement on F-measure compare TransDETR. This is due the domain gap between image data and video data, directly using image spotter leads to low confidence and consequently low Recall on video data. When encompasses the rescoring mechanism, GoMatching achieves a 9.1 improvement on Recall compared to DeepSolo and 6.5% F-measure enhancement compared to TransDETR. These improvements highlight the effectiveness of rescoring mechanism in alleviating the domain gap and leading to a better tracking candidate pool. The impressive results of GoMatching are not merely attributed to the introduction of a robust image text spotter.
To further explore how the rescoring mechanism eases the domain gap, we calculate the AP of detection results on ICDARI3-video, as shown in the Table 9. Incorporating the rescoring mechanism effectively improves the detection performance of DeepSolo on video datasets, particularly for small text, resulting 3.9% improvement APs. We also present more visual results to embody the potency of rescoring mechanism in Fig. 8.
G More Visualization Results
We present more visualization results of TransDETR and GoMatching in Fig. 9, including some failure cases. It can be observed that GoMatching exhibits significant improvement in text recognition performance compared to TransDETR. It should be noted that GoMatching may also experience failures due the image-viee video domain gap and extreme cases, such very small text instances and significant motion blur. These issues can be mitigated by employing a stronger text spotter with more representative backbone and training larger, more diverse datasets.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of GoMatching in Appendix G.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed information about the training and inference settings in Section 4.2 and Appendix B, along with the release of the code and trained models, ensuring that all necessary information for reproducing the main experimental results is disclosed.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided the data and code, with sufficient instructions for open access in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
(eglli//gidee/Codeeeee
irriiiissoooPoiiiii onPolicy) icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. cctpublic//aeccccccccccccccc /gwpi/gg///////eeee// /Ce///eeembbiiic cttsSSubbission.. ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify the training and test details in Section 4.2 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
The experiments reported in the paper do not include error bars. Instead, focus on providing comprehensive results and analyses to demonstrate the performance of our approach.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide relevant information in Section 4.2 and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: btetps://neurtii /nwurisuu...p// cc
Answer: [Yes]
Justification:
No ethical issues involved.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our paper aiming to improve performance and efficiency text spotting from videos, and does not have any direct negative applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All data and models utilized in this paper are publicly available and appropri- ately cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We are submitting the code for GoMatching in the supplementary material and providing a "readme.md" file to reproduce our results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "ZM8kXtbaOB", "reviewer": "Reviewer_1HqK", "summary": "The paper describes an approach for text spotting in videos. The method uses a text spotting method for still images to detect text instances frame by frame and the output is further processed by a specific module to find association between text instances through adjacent frames. In addition, a new dataset for video text spotting with curved text intances is presented. Experimental results are given for this new dataset and for several standard benchmarks for video text spotting.", "questions": "- The base DeepSolo method is kept frozen during training. What is the reason for that? Have you tried training also this part and see waht results are obtained? Some analysis and discussion about this would be interesting", "limitations": "There is no specific discussion on the limitations of the method", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes a simple approach to get the associations between text instances in different frames obtained with a standard text spotting method in single images. Although I am not very familiar with archtectures for video text spotting, the cntribution seems interesting and it seems to have the potential to be used with different image text spotting methods.\n- Experimental results show that the proposed method performs well and also that the different components of the approach contribute positively to the global performance of the method.", "weaknesses": "The paper is based on two main claims: one about the gap between detection and recognition and potential problems with optimization of association and the other, the lack of curved text in training data. I do not see through the paper how the proposed method and dataset help to solve these problems. Experimental results are good, but they are not put in relation with the initial claim and it is not shown how the proposed method help to alleviate this specific problem. \nOn the other hand, a new dataset with curved text is proposed, but this dataset is not used to enrich the training data in the general setting and thus, it is not shown how it alleviated the lack of training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thoughtful and insightful comments. Below we address the key concerns and promise to incorporate all feedback in the revised version.\n\n**Q1. How the proposed method and dataset help to solve the gap between detection and recognition, the optimization conflict, and the lack of curved text in training data?**\n\n**A1:** In this paper, we started by analyzing the bottleneck of the SOTA video text spotter (TransDETR) and found its **inferior recognition ability, particularly on curved text**. While comparing VTS to ITS methods which have substantially promoted the recognition ability, we summarize two key aspects of the detection-recognition gap: 1) **the inferior model architecture**, particularly the recognition part, and 2) **the lack of  diversity in video data**. However, collecting large-scale video training data with diverse curved text is extremely **time- and labor-consuming**. Thus, **we are motivated to resort to the off-the-shelf recognition model of the SOTA image text spotter (e.g., DeepSolo) and leverage its excellent capability in text recogniiton, especially for curved text recognition**. We find that DeepSolo already delivers superior recognition performance on video data in zero-shot setting (Fig. 1). We propose to efficiently turn an image spotter (cannot conduct tracking) to a video spotter. ***By leveraging a frozen DeepSolo, we not only promote the video text recognition ability but also preserve the knowledge from diverse image data, indirectly alleviating the lack of curved text in video. Additionally, keeping DeepSolo frozen prevents potential optimization conflict between text spotting and tracking, and helps focus training efforts more on tracking***. Subsequently, we introduce a rescoring head to better adapt DeepSolo to the video domain at low cost, and design the LST-Matcher to cast the strong tracking ability. Finally, the substantial improvements on extensive video benchmarks support these effective designs.\n\nAs for our proposed ArTVideo, it is used for evaluating the video text spotting on curved text, filling the gap in video field for the first time and facilitating subsequent research. In future work, we can try to leverage several foundation models and SOTA specialists to filter and label a high-quality, large-scale, and diverse video text dataset, directly solving the lack of training data of video curved text.\n\n\n**Q2. What are the reasons for keeping DeepSolo frozen during training?**\n\n**A2:** There are three key reasons. **1) Preserving the valuable knowledge learned from diverse image data**. As shown in Fig. 1(a), we found DeepSolo has superior recognition performance in zero-shot testing on video data, particularly for curved texts. When we unfroze DeepSolo on video data, we observed an unexpected performance drop, as demonstrated in the table in **A3**. This indicates that there are knowledge forgetting or overfitting issues when fine-tuning on video data with relatively low text shape and vocabulary diversity. Thus, to better preserve the knowledge learned from diverse image data, we keep it frozen. **2) Saving computational resources**. Compared to image text spotting, video text spotting requires additional temporal-level learning. On each GPU, GoMatching needs 6 frames to learn temporal relationships. Unfreezing DeepSolo would lead to more computational overhead and higher GPU memory consumption. For example, unfreezing all parameters of DeepSolo would cause an unaffordable out-of-memory issue. In contrast, freezing DeepSolo only requires 3 GPU hours of training on ICDAR15-video using one 3090 GPU. **3) Focusing more on tracking optimization**. As video text spotting involves three sub-tasks, keeping DeepSolo frozen can prevent potential optimization conflicts between text spotting (detection and recognition) and tracking.\n\n\n\n**Q3. Have you tried training also DeepSolo?**\n\n**A3:** Thank you for your suggestion. Actually, in Appendix F, we have discussed the influence of different training strategies (including unfreezing DeepSolo) on ICDAR15-video. The results are shown below:\n\n| Index |                Training Setting                | MOTA ($\\uparrow$) | MOTP ($\\uparrow$) | IDF1 ($\\uparrow$) |\n| :---: | :--------------------------------------------: | :---------------: | :---------------: | :---------------: |\n|   1   |        'Only LST-Matcher' (**default**)        |     **72.04**     |       78.53       |     **80.11**     |\n|   2   | First 'Only DeepSolo', Then 'Only LST-Matcher' |       70.82       |       78.09       |       79.64       |\n|   3   |   'End-to-End', DeepSolo's Decoder ('0.001')   |       71.48       |     **79.14**     |       78.98       |\n|   4   |   'End-to-End', DeepSolo's Decoder ('0.01')    |       70.15       |       78.17       |       77.67       |\n|   5   |    'End-to-End', DeepSolo's Decoder ('0.1')    |       68.03       |       75.46       |       77.16       |\n\nIn the table, 'Only DeepSolo' and 'Only LST-Matcher' respectively refer to fine-tuning of DeepSolo and LST-Matcher while keeping other modules fixed. 'End-to-End' denotes training both DeepSolo and LST-Matcher in an end-to-end manner. '0.001', '0.01', and '0.1' represent the ratios of the DeepSolo decoder learning rate relative to the base learning rate. Due to constraints in training resources, for DeepSolo, we only unfroze its decoder during end-to-end training. \n\nWe observe that fine-tuning DeepSolo on the video dataset (row 2) leads to performance decline compared to the default setting (row 1), caused by the low diversity and quality of the video data.  Moreover, when training GoMatching end-to-end  (row 3 to 5), the performance gradually declines as the decoder's learning rate increases. This is likely due to conflicts among tasks. Therefore, it would be worthwhile to explore more effective multi-task optimization strategies in future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tGVb9Kx3Y5", "reviewer": "Reviewer_Lmjj", "summary": "This paper adopts the idea of tracking-by-detection and apply it in the task of video spotting. The proposed algorithm is built on top of a SOTA image text spotting model and the authors contributions lie in the tracking part. They design LST-Matcher to integrate both short-term and long-term matching results. Experiments on multiple real datasets show that the proposed method outperforms state-of-the-art video spotting approach.", "questions": "Please address the concerns in W1, W2, W3 and W4.", "limitations": "The authors failed to discuss the limitations of the work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. Applying the idea of tracking-by-detection in the task of video spotting is interesting and useful.\n\nS2. Experimental results are impressive.", "weaknesses": "W1. The authors adopted a powerful image text spotting model Deepsolo and compared it with TransDETR. This may cause an issue that requires further investigation, i.e., how much performance improvement was brought by Deepsolo? In multi-object tracking, the performance of object detector plays an important role to the overall performance.  Therefore, it becomes difficult to judge contributions brought by LST-Matcher.\n\nW2. Multi-object tracking is a very crowded research area. Obviously, the literature review in this paper missed a lot of recent works. My question is since there are so many works belonging to the category of tracking-by-detection, why the authors eventually choose the technical solution as in LST-Matcher? Can we adopt the idea of other works such as ByteTrack or its subsequent works? I noticed that TencentOCR incorporated ByteTrack and demonstrates better performance than GoMatching.\n\nW3. Following W2, TencentOCR exhibits better video text spotting results. So, what is the drawback of TencentOCR in real practice. Is it less efficient than GoMatching?\n\nW4. In TransDETR, detection results are also reported. It would be better to report the quality of detection task. Even though detection is not part of contributions in this paper, it still can help explain W1 to see the contributions brought by Deepsolo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your positive and insightful comments. Below we address the key concerns and promise to incorporate all feedback in the revised version.\n\n**Q1. How much performance improvement was brought by DeepSolo? It would be better to report the quality of detection task.**\n\n**A1:** Thanks for your suggestion. Following TransDETR, we provide the detection performance on ICDAR13 as follows:\n\n|               Method                |  Precision   |   Recall    |  F-measure  |\n| :---------------------------------: | :----------: | :---------: | :---------: |\n|              Free [1]               |     79.7     |    68.4     |    73.6     |\n|            TransDETR [2]            |     80.6     |    70.2     |    75.0     |\n| GoMatching w/o rescoring (DeepSolo) | 92.4 (+11.8) | 65.7 (-4.5) | 76.8 (+1.8) |\n|             GoMatching              | 89.5 (+8.9)  | 74.8(+4.6)  | 81.5 (+6.5) |\n\nWithout the rescoring mechanism, GoMatching outputs the original zero-shot results of DeepSolo, resulting in a 4.5% decrease in Recall and only a 1.8% improvement in F-measure compared to TransDETR. This is due to the domain gap between image data and video data. For detection, directly adopting a frozen image text spotter leads to low confidence and consequently a relatively low Recall on video data. DeepSolo mainly promotes the recognition performance, especially for curved text, as evidenced in Fig. 1(a) of the main paper. Using the rescoring mechanism, GoMatching achieves a 9.1% improvement in Recall compared to DeepSolo and a 6.5% F-measure enhancement compared to TransDETR. These improvements validate the effectiveness of the simple rescoring mechanism in alleviating the domain gap and leading to a better tracking candidate pool. The impressive results of GoMatching are not merely attributed to the introduction of a robust image text spotter.\n\n>[1] Free: A Fast and Robust End-to-End Video Text Spotter. IEEE TIP, 2020.\n>[2] End-to-End Video Text Spotting with Transformer. IJCV, 2024.\n\n\n\n**Q2. Why authors choose the technical solution as in LST-Matcher? Can we adopt the idea of ByteTrack?**\n\n**A2:** Thanks for your thoughtful question. There are two key aspects: 1) performance, and 2) simplicity. From the perspective of performance, when replacing the LST-Matcher with ByteTrack, the MOTA significantly decreases from 72.04% to 65.05%, demonstrating the superior tracking capability of LST-Matcher. In addition, LST-Matcher only contains two Transformer encoder-decoder blocks, consuming little computational resource and inference time. We provide the proportion of inference time for each component on ICDAR15-video in the following table. Within the whole pipeline, LST-Matcher only consumes 5.43% of the inference time.\n\n| DeepSolo | Rescoring Mechanism | LST-Matcher | Other (pre/post-processes) |\n| :------: | :-----------------: | :---------: | :------------------------: |\n|  87.78%  |       <0.01%        |    5.43%    |           6.79%            |\n\n\n\n**Q3. What is the drawback of TencentOCR in real practice?**\n\n**A3:** TencentOCR is a solution with remarkable performance for the DSText competition. However, as we described in Sec. 4.3 of the main paper, it **ensembles** the results of **several models** with **multiple backbone architectures**. DBNet and CasCade MaskRCNN are employed for detection, while Parseq and ByteTrack are adopted for recognition and tracking, respectively. Ensembling the results of the integrated pipeline under different backbone settings **requires more deployment and computational cost**, limiting the real practice, **particularly for real-time response**. Besides, the model weights of TencentOCR are not released. In contrast, as shown in Tab. 1(c) of the main paper and Tab. 5 in the supplementary material, our GoMatching outperforms TencentOCR in terms of MOTA on DSText (22.83% vs 22.44%)  with a simpler structure and inferences faster than TransDETR (14.41 FPS vs 12.69 FPS on ICDAR15-video in the same image size setting). It demonstrates that GoMatching is a simple, strong,  yet efficient baseline for video text spotting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m03QukLNlC", "reviewer": "Reviewer_xYhx", "summary": "This work uses a query-based image text spotter for video text spotting to solve the poor recognition issue. To achieve this, they add a rescoring head to restore the confidence of detected text instances and use transformers to enhance the tracking capability in videos.", "questions": "1. Lack of motivation.\n2. what is the domain gap between an image and a single video frame?\n3. How to tackle multilingualism?", "limitations": "No there is no explicit mention of the limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Extend an image text-spotting mechanism to a video text-spotting scenario.\n2. Introduce a rescoring mechanism to mitigate the domain gap between image and video datasets.\n3. Long-short-term matching strategy to enhance the tracking capabilities.\n4. An extensive experimental study has been provided.", "weaknesses": "1. It is a generic comment that the recognition text in all the qualitative examples is not visible even with zooming in. Also, in the video they have been provided in the supplementary, I think there are some arbitrary shape text misrecognized. The description of the methodology section is too confusing. Thanks to the author for providing the code as supplementary to help me understand the work.\n\n2. The input to the Image Text Spotter (i.e. DeepSolo) is a single frame, not multiple frames. Then what is the domain gap here? How the proposed rescoring algorithm solves the domain gap. I think this rescoring mechanism is for highlighting the small text.\n\n3. It is also not clear to me how this technique works for multi-language settings (i.e. English and Chinese). In Deepsolo its completely dataset dependent. But here in Figure 4, in a single frame, the model can detect English as well as Chinese text. The mechanism of this multilingual OCR is not mentioned in the paper.\n\n4. The motivation for expanding an image text spotter to a video text spotter is not clear to me. Especially when you are working with a single frame. On the other hand, there is a recent work on \"End-to-End multi-view scene text recognition\" (Pattern Recognition, 2024) which considers three views together to enhance the recognition performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thoughtful and insightful comments. Below we address the key concerns and promise to incorporate all feedback in the revised version.\n\n**Q1: About the recognition text in qualitative examples and the description of the methodology.**\n\n**A1:** Thanks for your pointing out. We will enlarge the recognized texts to make them clearly visible, as shown in the provided qualitative results in the rebuttal PDF. We will also endeavor to revise the methodology section and make it easier to understand, such as adding more illustrations and more explanations for the equations.\n\n\n\n**Q2. What is the domain gap? How the proposed rescoring algorithm solves the domain gap?**\n\n**A2:**  The domain gap between video and image arises from differences in form and data source. Specifically, video frames often contain and concern more small and blurry text instances caused by camera distance and motion. In our method, DeepSolo trained on image data is employed and kept frozen. We found that the frozen DeepSolo tends to provide lower confidence for text instances in video data, leading to sub-optimal performance. Adopting the simple rescoring mechanism can help distinguish text instances through confidence calibration and prevent some instances from being filtered out by the confidence threshold, thus better adaptation to unseen video data.\n\nTo further validate above analysis and show how rescoring head works, we provide the detection F-measure and AP results on ICDAR13-video. ICDAR13-video is selected because of its available testing GT for calculating AP. As can be seen, using rescoring achieves 4.7% improvement on F-measure, mainly resulting from the large enhancement on Recall by 9.1%. Considering the AP results, rescoring mainly improves the AP_S metric (AP for small instances with areas less than 32\\*32 pixels) . These evidences validate that the rescoring mechanism solves the domain gap by calibrating the confidence score and avoiding some small and blur texts from being filtered, leading to a better tracking candidate pool. We also provide qualitative comparisons in the attached PDF for visualization support. We agree that the rescoring mechanism can highlight small texts. It can also highlight the instances with medium and large size, as demonstrated by the 2.6% and 2.1% enhancements on AP_M (AP for medium instances with areas between 32\\*32 pixels and 96\\*96 pixels) and AP_L (AP for large instances with areas larger than 96\\*96 pixels).  \n\n| Method | Precision | Recall | F-measure |\n| :---: | :---: | :---: | :---: |\n| TransDETR | 80.6 | 70.2 | 75.0 |\n| GoMatching w/o rescoring | 92.4 | 65.7 | 76.8 |\n| GoMatching | 89.5 (-2.9) | 74.8 (+9.1) | 81.5 (+4.7) |\n\n\n| Method | AP | AP_S | AP_M | AP_L |\n| :---: | :---: | :---: | :---: | :---: |\n| w/o rescoring | 26.2 | 11.6 | 40.1 | 49.8 |\n| w/ rescoring | 29.3 (+3.1) | 15.5 (+3.9) | 42.7 (+2.6) | 51.9 (+2.1) |\n\n\n\n**Q3: It is also not clear to me how this technique works for multi-language settings (i.e. English and Chinese).**\n\n**A3:** DeepSolo released both English-only and bilingual (English and Chinese) version model weights. For bilingual video text spotting on BOVText, we use the officially released bilingual version DeepSolo weights instead of the English-only version. We do not introduce extra techniques in the rescoring mechanism and LST-Matcher. As detailed in Tab. 1(b), our GoMatching achieves a 41.5% improvement on MOTA compared to the previous SOTA method on bilingual BOVText dataset, further underscoring the scalability and potential of our proposed baseline for multilingual text spotting. In the future, one could further expand the character table of the text spotter or explore Mixture-of-Experts to assemble different recognizers for different languages. Our method has the potential to seamlessly transfer the multilingual image text spotter into multilingual video text spotter at low cost.\n\n\n\n**Q4: The motivation for expanding an image text spotter to a video text spotter is not clear to me.**\n\n**A4:** In this paper, we focus on video text spotting (VTS) which requires text detection, recognition, and tracking across *multiple frames*. We observe that the SOTA VTS model exhibits inferior text recognition proficiency on oriented and curved text, limiting the real-world applications. These bottlenecks have not been identified by previous VTS methods. In the static image realm, image text spotters (which cannot perform tracking) have significantly improved recognition ability. We compare video methods with image methods, and summarize two key aspects for existing VTS models: 1) the inferior model architecture, particularly the recognition part, and 2) the lack of diversity in training data, such as the inclusion of few arbitrarily-shaped texts. However, collecting and annotating large-scale video training set with abundant curved texts is extremely costly. Thus, we propose to efficiently turn an image text spotter to a video text spotter, leveraging the merits of off-the-shelf architecture and the valuable knowledge learned from diverse image data. Subsequently, we design a simple rescoring mechanism to adapt the image text spotter to video domain at low training cost, and introduce LST-Matcher to cast the strong tracking ability. Please also refer to the response **A1** to Reviewer 1HqK.\n\nE2EMVSTR [1] offers a novel and practical method for enhancing scene text recognition. E2EMVSTR inspires us to improve video text spotting by mining the semantic consistency across frames and better distinguishing different tracklets. We will cite this paper and add discussion on potential improvements.\n\n>[1] An End-to-End Model for Multi-View Scene Text Recognition. PR, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jrGTeeY4Br", "reviewer": "Reviewer_BgWx", "summary": "The paper introduces GoMatching, a streamlined and efficient baseline for video text spotting that enhances tracking capabilities through a novel Long-Short Term Matching module, while also setting new performance benchmarks on multiple datasets and introducing the ArTVideo test set for arbitrary-shaped text evaluation.", "questions": "see weaknesses", "limitations": "The reviewer did not see any discussion of limitations in the main text", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This method leverages an off-the-shelf, query-based image text spotter and adapts it to video datasets through a focused training regimen on tracking while maintaining robust recognition performance.\n\n2. A key component of GoMatching is the LST-Matcher, which enhances tracking capabilities by integrating both long-term and short-term matching results through a Transformer architecture.\n\n3. The paper reports that GoMatching sets new records on multiple benchmark datasets, including ICDAR15-video, DSText, BOVText, and a newly proposed test set called ArTVideo, which features arbitrary-shaped text.", "weaknesses": "1.  The paper does not provide a corresponding share of inference time; the reviewer recommends that an analysis could be presented and compared with previous work (TransDETR). It is not a problem if the inference speed has not achieved state-of-the-art; such analysis can further support the paper.\n\n2. The related ablation study only presents simple numerical comparisons of experimental results, which appear overly simplistic. More detailed analysis is necessary. It would be better if corresponding visual results comparisons could be provided if possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your positive and insightful comments. Below we address the key concerns and promise to incorporate all feedback in the revised version.\n\n**Q1: An analysis of inference time could be presented and compared with previous work (TransDETR).**\n\n**A1:** We included the comparison of inference FPS in Tab. 5 of the supplementary material, as shown in the following table. \"Size\" denotes the shorter side of the input image during inference. All FPS results are measured on one 3090 GPU. With the same image size setting as TransDETR, GoMatching achieves faster FPS while obtaining significant performance improvements over all metrics, particularly on MOTA. While resizing the shorter side to 1000, GoMatching outperforms TransDETR by 11.08 MOTA at the cost of only 2.09 lower FPS. **Overall, GoMatching can get significantly better performance with faster FPS.**\n\n|         Method          | MOTA  | MOTP  | IDF1  |  FPS  |\n| :---------------------: | :---: | :---: | :---: | :---: |\n|  TransDETR (Size: 800)  | 60.96 | 74.61 | 72.80 | 12.69 |\n| GoMatching (Size: 800)  | 68.51 | 77.52 | 76.59 | 14.41 |\n| GoMatching (Size: 1000) | 72.04 | 78.53 | 80.11 | 10.60 |\n\n\n\n**Q2: The related ablation study only presents simple numerical comparisons of experimental results, which appear overly simplistic. More detailed analysis is necessary. It would be better if corresponding visual results comparisons could be provided if possible.**\n\n**A2:** Thank you for your valuable advice. In the supplementary material, we provided the visual results comparison between ST-Matcher and LST-Matcher in Fig. 8. We will add more analysis as shown below. With ST-Matcher which only leverages short-term frames, the image blur due to camera motion would cause strong appearance changes and corresponding feature variation, probably leading to the tracking candidate mismatch and ID switch issue. In contrast, by aggregating long-term information which is overlooked in ST-Matcher to alleviate the influence of short-term appearance changes, LST-Matcher can better address the ID switch issue.\n\nAdditionally, we add a visual comparison between with and without rescoring mechanism in the attached PDF. Without rescoring, the frozen text spotter from static image domain tends to convey low confidence on small and blur texts caused by camera motion. Adopting rescoring mechanism helps better adaptation to unseen video datasets, thereby distinguishing these text instances via confidence calibration and avoiding them from being filtered out by the confidence threshold.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their insightful reviews and kind support for our work. We are encouraged that the reviewers appreciate the interesting contribution and idea (Reviewer 1HqK, Lmjj), the good and impressive results (Reviewer 1HqK, Lmjj, BgWx), and the potential for wider usage with different image text spotting methods (Reviewer 1HqK).

We provide detailed responses to each reviewer respectively, and we promise to incorporate all feedback in the revised version.


**Explanation of supplementary PDF:**

Regarding the supplementary PDF, we provide larger and clearer visual results of GoMatching and further analysis the effectiveness of our proposed method,  following the suggestions from Reviewer xYhx and BgWx.

Meta-review: The mansuscript received mixed reviews. The authors' response added new information and helped clarify many of the outstanding issues raised by the reviewers.

Overall, the reviewers agree that the contribution is technically sound. The experimental results, and new results presented during rebuttal, are good and convincing, although I agree with reviewer 1HqK in that they could be better put in relation with the initial claims in the introduction.

I agree with the authors that there is typically a non-negligible domain gap between static scene text images and video frames due to encoding, motion blur and a number of other effects.

Overall, taking into account the reviews, and the lengthy discussion between the authors and reviewers, I consider that the paper is worth presenting in the conference, and encourage the authors to incorporate the key points of this discussion / reviews in the final manuscript.
Decision: Accept (poster)