PBFormer: Capturing Complex Scene Text Shape with Polynomial Band Transformer
Authors: Ruijin Liu
Ning Lu
Dapeng Chen
Cheng LI
Zejian Yuan
Wei Peng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Complex Shape Text Detection
Text Representation
Transformer
Computer Vision
Application

Abstract: We present PBFormer, an efficient yet powerful scene text detector that unifies the transformer with a novel text shape representation Polynomial  Band (PB).  The representation has four polynomial curves to fit a text's top, bottom, left, and right sides, which can capture a text with a complex shape by varying polynomial coefficients.  PB has appealing features compared with conventional representations: 1)  It can model different curvatures with a fixed number of parameters, while polygon-points-based methods need to utilize a different number of points.  2) It can distinguish adjacent or overlapping texts as they have apparent different curve coefficients, while segmentation-based methods suffer from adhesive spatial positions. PBFormer combines the PB with the transformer, which can directly generate smooth text contours sampled from predicted curves without interpolation.  To leverage the advantage of PB,  PBFormer has a parameter-free cross-scale pixel attention module.  The module can enlarge text features and suppress irrelevant areas to benefit from detecting texts with diverse scale variations.  Furthermore, PBFormer is trained with a shape-contained loss, which not only enforces the piecewise alignment between the ground truth and the predicted curves but also makes curves' position and shapes consistent with each other.  Without bells and whistles about text pre-training, our method is superior to the previous state-of-the-art text detectors on the arbitrary-shaped CTW1500 and Total-Text datasets. Codes will be public.

Under review as a conference paper at ICLR 2023
PBFORMER: CAPTURING COMPLEX SCENE TEXT SHAPE WITH POLYNOMIAL BAND TRANSFORMER
Anonymous authors Paper under double-blind review
ABSTRACT
We present PBFormer, an efficient yet powerful scene text detector that unifies the transformer with novel text shape representation Polynomial Band (PB). The representation has four polynomial curves to fit text's top, bottom, left, and right sides, which can capture a text with a complex shape by varying polynomial coefficients. PB has appealing features compared with conventional representa- tions: 1) It can model different curvatures with a fixed number of parameters, while polygon-pon Selyygon-points--aeee points-based methods need to utilize different number of points. 2) It can distinguish adjacent or overlapping texts as they have apparent different curve coefficients, while eegmentation-basee methods suffer from adhesive spatial positions. PBFormer combines the PB with the transformer, which can directly generate smooth text contours sampled from predicted curves without interpola- tion. A murameter-free cross-scale pixel attention (CPA) module is employed to highlight the feature map of suitable scale while suppressing the other feature maps. The simple operation can help detect small-scale texts and is compatible with the one-stage DETR framework, where no postprocessing exists for NMS. Furthermore, PBFormer is trained with shape-contained loss, which not only en- forces the piecewise alignment between the ground truth and the predicted curves but also makes curves' position and shapes consistent with each other. Without bells and whistles about text pre-training, our method is superior to the previous ttate-of-the- art text detectors the shaape ETW1500 and Total-Text datasets. Codes will be public.
INTRODUCTION
Scene text detection is an active research topic in computer vision and enables many downstream applications such image/video understanding, visual search, and autonomous driving (Radford) jtt al. 2021] Long et al. 2021 Reddy et al. 2020). However, the task also challenging. One non- negligible reason is that the text instance have complex shape due to the non-uniformity of the text font, skewing from the photograph, and specific art design. Capturing complex text shapes needs to develop effective text representation. State-of-the-art methods roughly tackle this problem with two types of representations. One is the point-based representation, which predicts the points on the image space to control the shape of the points, including the Bezier control points (Liu al., 2020) and polygon points (Zhang et al. (02T). The other produces segmentation maps. The map can describe the text various shapes and can benefit from the prediction results at the pixel level Liao et al. 2020 Zhu et al. 2021b).
Despite the good performance, both types of representation have limitations: 1) Points- based meth- ods suffer from a fixed number of control points (Tang et al. 2022; Zhang al. 2022b). Too few points cannot handle the highly-curved texts, while simply adding points will increase redundancy for most perspective texts. 2) Segmentation-based methods frequently fail in dividing adjacent texts due to ambiguous spatial positions. The produced segmentation map still needs post-processing and often requires extensive training data (Zhu et al., 2021b).
To address these limitations, we propose a novel representation, named Polynomial Band (PB). has clear advantages compared with previous text representations. In particular, PB consists of four polynomial curves, each of which fits along a text's top, bottom, left, and right sides. First, the coefficients of PB discriminative in the parameter space even if the two texts very close in

---

Under review as a conference paper at ICLR 2023
WOODDWOOD
PLANET
y- 72 2-0.991x66.729
NET PL
HOLLYWOON
HOLZYWOOD
Pozza Nander Expinion
Y 21 PLANNTT botten
Voyyng Quidertic Caave Regrooxing Skaps fioe. Steige Highl Curved
TESTR(Borkc))
Seperation by
Figure 1: Advantages of PB. Comparing (a) and (b), PB divides adjacent texts more clearly than Bezier control points. (c) shows the number of output point increase gradually to represent shapes from straight to highly curved. (d) shows varying curve coefficients can handle dynamic shapes with two boundary variables.
the image space, as shown in Fig. Second, PB is represented by the functions defined in the image space. We can directly compare the ground truth contour points with the sampled points from polynomial curves by re-sampling techniques. It differs from Bezier-curve-bie basee methods that need to generate the intermediate representation, i.e., "control points" for supervision. The loss defined by the control points cannot truly reflect how humans percept the shape. A small difference in control points will lead to a large shape difference.
Witnessing the great success in NLP (Vaswani et al. 2017), there has been recent surge of interest in introducing transformers to vision tasks, including scene text detection. The current transformer- based text detectors are with et al. 2022b). We apply the proposed PB to the one-stage deformable DETR to improve the effi- eiency. In particular, we design parameter- free cross-scale pixel attention (CPA) module between the CNN feature and the transformer encoder-decodee layers. The CPA module first aligns the fea- ture maps of different scales by enlarging all the feature maps to the same scale. Then it performs the cross-scale attention that highlights the feature value from suitable scale while suppressing the other feature maps. With the scale-selective mechanism, our method becomes more compatible with the transformer decoders that do not have NMS for postprocessing. It implicitly suppresses the text proposals with incorrect scales, alleviating the learning burden of the transformer encoder- decoder layers. The features from CPA effective to represent the shape of the text, two layers of transformer encoder-decoders are sufficient to detect the reasonable size of PB without NMS.
stages, such FewBetter (Tang al.. 2022) and TESTR (Zhang)
The transformer decodes each polynomial curve's K coefficients and 2 boundary variables that determine the curve's definition domain. We uniformly sample the points on the predicted curves within the definition domain and compare them with the corresponding points on the ground truth polygon. Such a design supervises the curve piece by piece and can learn the curve shape and range consistently. In summary, the contribution of PB is:
A novel text representation called the Polynomial Band (PB) is proposed. PB can utilize a fixed number of parameters to capture the text instance with various curvatures. also excels at distinguishing the spatially close text instances.
A cross-scale pixel attention module is proposed. The module performs pixel-wise atten- tion across the feature maps with different sizes. It implicitly highlights the text regions and enables the transformer to direct take all pixel-wise features input.
We design a shape-constrained loss function. The loss enforces the piece-wise supervision over the predicted curve and consistently optimizes the curve coefficients and definition domains.
Experiments multi-oriented and curved text detection datasets CTW1500 (Liu et al. 2019 and Total-Text (Chng & Chan 2017) demonstrate the effectiveness of our approach. Without any pre- training on large-scale text datasets, method can achieve better results in terms of F-measure. Due to the lightweight network architecture, our method runs real-time and 4.4 x faster than other open-sourced transformer text detectors.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Text representation. Conventional text representation can be roughly divided into point-based and segmentation-basse methods. With the need to capture more complex text shapes, the point- based methods gradually involve more points for text representation, including 2 points bound- ing box (Li al. 2017), 4 points quadrilateral (Li et al., 2019 Sun et al. 2018 16 points poly- gon (Zhang et al. 2020) or 30 points polygon (Zhang et al. 2020) changed by texts' length. ABC- Net (Liu al.) 2020) calculates Bezier control points (8 points), which is sufficient for most quadri- lateral or slightly- curved texts but still suffers from highly-curved, One deficiency of point-based methods that they usually predict a fixed number of points, which hard to balance the per- formance between simple perspective texts and text instances with complex shapes (Zhang et al.) 2022b). ESIR (Zhan Lu 2019) adopts a single polynomial that fits the text center line to rectify the image for recognition. However, they can only model horizontal texts and still need points to represent text contours. contrast, the polynomial curves in polynomial band can handle various orientations and shapes, from straight line to round curve with fewer and fixed number of parameters.
Segmentation- based representation scription (Wang et al., 2019a; Liao et al., 2021; Wang et al., 2019b, Tian et al.) 2019 Zhhu et al.) 2021b Liao al.] 2020 2022). However, they frequently fail to divide adjacent texts due to am- biguous spatial positions. Although CentripetalTexx (Sheng et al., 2021) tackles this problem by detecting a shrunk text mask and reconstructing the contour by shift map, it suffers from high com- putation complexity. The proposed polynomial band overcomes this problem by considering the global curve shape. Curve coefficients in parameter space can be easily separated even though two texts are close, making more clear bounds in the text crowding scenario.
naturally handle complex-shane texts due to pixel-wise de-
Text detection transformer. There a trend to equip the transformer Vaswanni et al. 2017) with scene text detection. Current methods (Tang et al. 2022 Zhang al., (022b) directly combine DETR variants (Carion et al., 2020 Zhu et al. 202 with point representation such as 16-point polygons Bezier control points, and adopt the two-stage architecture for the ease of optimization. For example, FewBetter (Tang et al. (2022) first extracts segmentation maps by CNN-FPN (Lin et al. 2017) to show representative text regions, then samples feature points in each region and feed them into a transformer to further decode control points. TESTR (Zhang et al.) 2022b) follows (Wang al. 2020b to detect the bounding boxes, then utilize transformer find the control points in each bounding box. They sacrifice efficiency due to the two-stage pipelines and destroy the sim- plicity of the detection transformer scheme. Our PBFormer inherits the single-stage simplicity and inserts parameter- free cross-scale pixel attention module between the CNN feature and transformer encoder-decoder layers. With the attention module, per-pixel features are fed into the subsequent transformer without generating any proposals, making the whole architecture more efficient.
3
METHODOLOGY
The overall framework of PBFormer is illustrated in Fig. 2 Given an image with texts, PBFormer first employs a ResNer50 produce the multi-scale feature maps, then feed the feature maps cross-scale pixel attention module to highlight the texts' context information. The enhanced feature maps are concatenated and fed utilizes four polynomial curves to represent the shape of the text instance. It is simple but effective to capture different forms of texts. In the training phase, we sample the dense points from the curves for PB parameters estimation according to both predicted curve coefficients and the domain variables. A shape-constrained loss is designed to supervise the curves piece by piece.
lightweight transformer, predicting the PBs' parameters. PB
3.1 NETWORK ARCHITECTURE.
The network contains three modules: ResNet-based CNN encoder, cross-scale pixel attention module, and a lightweight deformable transformer. We now introduce more details about the atten- tion module and the decoders in the transformer.
Cross-scale pixel attention. The motivation of cross-scale pixel attention is to highlight text fea- tures at the best scale and suppress the others. Such selective mechanism is compatible with
3

---

Under review as a conference paper at ICLR 2023
PR "DISTILLERY
SAEEETT PIRCENC LINOGNAEI TRSTERS BUISKUNN
x 1163, (723)
NB
ResNet50
Cross-scalc Pixel Attention
Lightweight DeformableTR
55 [PAR 2107)
ROMPANN BER
1885
ymu 12y dotyva
sepa (189)
Lightweight DeformableTTrnstoone
Cross-scale Pixel Attention
D.
Encoder Layer
Decoder Layer
Decoder Layer
DONGOW
Cruss scule Attention Mup Lincar Layers Figure 2: Architecture of PBFormer. The @ and + are element-wisely multiplication and addi- tion. respectively. In the lightweight Positional Embeddings (PE) are all learned parameters.
Object Queries PE Positional Embedding
the object queries (white box) and
DETR-like detectors that do not have non-maximum suppression. performs scale attention by comparing the values of the existing features among different scales with SoftMax. More details are illustrated in Fig. 2] the feature maps a square image from the ResNet backbone are with the size R1, R2, R3, R4- We enlarge them square feature maps with sizes D1, D2, D3. D4. Fol- lowing Deformable-Der DETR, the four feature maps are transformed to have the same channels by x convolutions. Then re-scaled them have the same size D and assemble them to obtain F E RDXxXCX4 After that, we use a SoftMax layer to compute the attention map for each pixel and each channel: iik softmax([F5jk) Figk2 Fisk3 F'jR41), i [1, jE D],
[1. CJ. The attention map A and the feature map F have the same shape, i.e., A, F RDxDxXx4 They multiply together to obtain the enhanced feature map F'. We disassemble F' to four feature maps. All of them are with the size D and the channel dimension C. We re-scale their size to be D1, D2, D3 and D4- It is noteworthy that the whole cross- cross-pixel pixel attention module is parameter-free, which brings training burdens during gradient back pack-propagation.
Lightweight deformable transformer. The four feature maps from the cross-scale pixel attention module are flattened to be vectors. We concatenate them long vector, then feed the vector to the deformable transformer. To predict the parameters of PB, we reduce the layers of the standard transformer encoder and decoder from 6 to 2, which is sufficient to yield competitive results. In deformable transformer, the reference points attend a small set of key sampling points nearby for each query, which are important to the deformable attention module. We adopt coarse-to-fine strategy to generate the reference points for the two decoder layers. In the first decoder, we adopt rough 2-d reference points derived from the positional embedding via linear projection. In the second decoder, we combine the same 2-d reference points with a 2-d vectors transformed from the output of the first decoder. In particular, the 2-d vectors encode the relative offsets according the first decoder's 's learned non-local dependencies, which help to generate more reasonable reference points for the second decoder layer. After that, a 3-layer MI.P generates the PB predictions over the entire image.
3.2 POLYNOMIAL BAND
We utilize four polynomial curves to represent the text instance's top, bottom, left, and right sides. The top and bottom boundaries represented by y f (a) and y fh
y = f (x) aرك²² af af E [ep e
(1)
y = ld (x) = aza² + + ab.
x e [ch b
4

---

Under review as a conference paper at ICLR 2023
where (z. y) is the coordinate of point on the boundary. a², aj aly. al, al. aß polynomial coefficients. [c): ef] and le ef] are range of a variable.
One critical problem is that the polynomial curve is single-value function which means point in the definition domain has a unique value the value domain. The functions of the curves along the horizontal direction cannot be used to represent the curves along the vertical direction. For example, as Fig. 2 shows, the text 'DISTILLERT's left (or right) side would not be represented by any y f (ar). We utilize - f (y) and x f (v) represent the left and right polynomial
curves of the text instance:
x= f (y) u2y² 2 + d ab
ye [c.
(2)
x x= f" (v) = ahy² + aiy + ab
ye lc.
where 1cb: e1] and [egs ef] define the range of y variable.
Output definition. We use four polynomial curves f² (x) y= xx) ( x=) (v)
that denote a band to wrap text instance. Thus, the output is 20-tuple 0 that consists of all polynomial coefficients and boundary variables in the form of:
0= (a aj, ab. cby ei, ab. ai ag: eū. ej, agy, a', dg. eb e ab aj og c ei)
where as, db, as + O, cb. E [0, 1], and all of them are real numbers.
3.3 LOSS FUNCTION
We leave the part about how to generate four ground truth point sets for the top, bottom, left and right sides from an original annotated polygon annotation in the Sec. [6.4] In this section, we introduce the shape-constrained loss to supervise the whole network. The network outputs N different PB parameters for each image, while their correspondences to ground truth contours are unknown. In this section, we first introduce how to compute the similarity between the predicted PB and ground truth contour by Shape-constraines loss, then provide the loss function for the whole image based on optimized correspondences solved by bipartite matching.
The shape constraints for each curve. We first revisit the curve fitting loss without constraints used the lane detection (Liu et al. 2021 2022). The ground truth fitting points of top or bottom curve are given by:
K 30 i,
(( 0.)
=
K
where the points are ordered from one end to the other, and the adjacent points for the top and bottom curves have the equal distance. The conventional fitting loss is:
(o)
Σ 11. - ( + lleo - 20ll1 + ||e1 - 2xK111 1=0
(5)
The fitting loss for the left and right curves can be obtained by exchanging the x and variables. However, such a fitting loss is unsuitable for detecting texts with diverse shapes and different posi- tions. It has two limitations: (1) the predicted curve segment is not aligned with the ground truth fitting points piece-by-piese
(2) the shape and range of the curve are independently optimized. As demonstrated in Fig. the conventional loss is not sensitive to the length of the curves, therefore the text detector tends to detect curves with inaccurate lengths.
We consider impose the shape constraints. The points on the predicted curve sampled accord- ing to both curve shape and range:
p f(x))) =; cu K
i.
Then compare the predicted points P and ground truth points p:
K
C(P,
P)
Σ
|
iilli + [[{(x)) - -Vall1-
(7)
1=0
Leveraging Eq. 7 in text detection encourages PB reconstruct the correct length of the contours.
5

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
(d)
GT Contour Points
Curve Segment
Errors
Evenly PRED Samplings
Figure 3: Diagram of shape constraints. All curve segments have the same curve coefficients. (a) and (c) have same ranges, so do (b) and (d). Without shape constraints, (a) and (b) show how to compare predicted curve segment with ground truth contour points. (c) and (d) illustrate the way with shape constraints.
The bipartite matching for the whole image. Let the network output of one image be H = {hy = (cj, y=) where Cj the confidence score indicating the possibility of a PB covering text and N is set to be larger than the maximum number of texts in image. After sampling the points on the four curves according to Eq. 6 H can be further represented by: H = { (c,.5 Pj. Pj, P))
For bipartite matching, we pad the ground truth set H with non-text instances have size N. The element having text instance is represented by hy (ē,, Pj, Pj PJ). PJ. In particular, PJ.P). P)., P5 are sampled points according to Eq. 4 while they are not need to be instantiated in the matching cost for non-text instances thus are set to be UJ. êg is set to be for the text and 0 for the non-text class. We formulate a bipartite matching problem to find an optimal injective function
g A + H. L.e. g(i is the index the PB assigned to fitting the i-th ground truth text:
N
g" arg min Σ cti (hy, + Cfocal (ej, j-1
(8)
g
where Le is the fitting loss and Cfocal is the focal cost. The fitting loss compares the predicted contour and ground truth contour by using the loss defined in Eq. 7
Lhe (hy. (g())) 1,.00 (C(P)) Ps ((G)) Ph +C(P) P Patn) (P)))) Ph()))
(9)
Then, the focal cost is defined as the difference between the positive and negative costs:
foood CoCn) = >o | (1 (c()))))) log Cg(z) + (1 - a) ( log - G
(10)
where and the hyper-parameter for the focal loss. is used to address the class imbalance, and y adjusts the rate at which easy examples are down-weighted. À adjusts the weight of the focal cost. The bipartite problem (Eq, ( can be efficiently solved by the Hungarian algorithm.
Overall Loss. With the optimized g", the overall loss function is given by:
N
Coverall
cfit cft (hi + [Sooal (ōgs (G55Co(())
(11)
#=1
where the Looal (eg, (1)) is the focal loss:
Clocal (cy,Cor = A 18:20 -1-c ()) log Cg-()) 18z=0 (1 - a) c)-() 1) log (1 Cg- c)-
(12)
a, À and y are the same with the ones in Eq. 10
4 EXPERIMENTS
Datasets. CTW1500 (Liu et al. 2019 a multi-oriented and curved scene text detection bench- mark containing 1,000 training and 500 testing images. Annotations are based on the text-line level with fixed fourteen points. The majority of text instances are curved. Total-Text (Chng Chan and curved scene text benchmark, while it consists of various text shapes such as multidirectional quadrilateral. has 1255 training images and 300 testing images. Each instance is annotated by ten point text-line.
2017) is an another

---

Under review as a conference paper at ICLR 2023
Detection results on CTW 1500 and Total-Text without pre-training on any text datasets. "Rep." denotes the method's output representation. "F", "Prec.", "Rec." represent F-measure, Pre- cision, and Recall. All the results are from their official codes and models. TESTR* means training without recognition branch.
Table 1:
CTW1500
Total-Text
Method
Rep.
F.
Prec.
Rec.
FPS
F.
Prec.
Rec.
FPS
PSENet
Seg
78.0
80.6
75.6
3.9
78.3
81.8
75.1
3.9
PAN
Seg
81.0
84.6
77.7
39.8
83.5
88.0
79.4
39.6
FCENet
Seg
85.1
88.1
82.3
2.7
85.8
89.3
82.5
2.9
ContourNet
Pts
83.9
83.7
84.1
3.8
85.4
86.9
83.9
3.8
TextBPN
Pts
84.0
87.7
80.6
12.1
86.9
90.8
83.3
10.6
TESTR*
85.1
88.4
82.1
7.3
85.3
89.7
81.2
6.9
TESTR
Pts
85.3
87.9
82.8
5.6
85.6
90.7
81.1
5.3
TESTR*
Bez
84.7
87.9
81.8
7.3
86.3
90.3
82.6
6.9
TESTR
Bez
85.1
88.0
82.4
5.6
86.6
91.3
82.4
5.3
PBFormer
PB
87.0
89.6
84.5
24.7
87.1
92.1
82.6
24.6
Table Detection results on CTW15500 and Total-Text with pre-training on text datasets. MLT, ST, ArT, and CST are abbreviations for MLT2017, SynthText, ArT 2019 and CurvedSynthTens datasets. C+M+T means using a combination of CST, MLT, and Total-Texx for pre-training.
CTWI500
Total-Teae -ext
Method
Rep.
Ext.
F.
Prec.
Rec.
FPS
F.
Prec.
Rec.
FPS
PSENet
Seg
MLT
82.2
84.8
79.7
3.9
80.9
84.0
78.0
3.9
PAN
Seg
ST
83.7
86.4
81.2
39.8
85.0
89.3
81.0
39.6
DB
Seg
ST
83.4
86.9
80.2
22
84.7
87.1
82.5
DB++
Seg
ST
85.3
87.9
82.8
26
86.0
88.9
83.2
28
TextRay
Pts
ArT
81.6
82.8
80.4
3.2
80.6
83.5
77.9
3.5
DRRG
Pts
MLT
84.5
85.9
83.0
85.7
86.5
84.9
TextBPN
Pts
MLT
85.0
86.5
83.6
12.2
87.9
90.7
85.2
10.7
TESTR*
Pts
C+M+T
86.6
90.8
82.8
7.3
86.2
92.4
80.7
6.9
TESTR
Pts
C+M+T
87.1
92.0
82.6
5.6
86.9
93.4
81.4
5.3
ABCNet
Bez
CST
81.4
84.4
78.5
6.8
84.5
87.9
81.3
6.9
PewBetter
Bez
CST
85.2
88.1
82.4
88.1
90.7
85.7
TESTR*
Bez
C+M+T
85.9
90.6
81.6
7.3
87.4
92.4
82.8
6.9
TESTR
Bez
C+M+T
86.3
89.7
83.1
5.6
88.0
92.8
83.7
5.5
PBFormer
PB
CST
88.0
90.6
85.4
24.7
88.1
93.2
83.5
24.6
We follow the standard metrics F-measures, recall, and precision to eval- uate the performance. prediction is considered as positive only when its IoU from the corresponding ground truth contour is larger than 0.5.
Evaluation Metrics.
Implementation Details. The input image size is set to be 800 x 800 for training and testing. Loss coefficient a, Y and À are as 0.25, 2 and 2. The fixed number of output N is 300. the cross- scale pixel attention module, Ri, R2, R3, R& are 100, 50, 25, 13, D1, D2. D3, D4 set as 128, 64, D2. For training from scratch, the learning rate set to be x 10-4 and
32, 16, and we set decayed ten times at 7200 epochs, and the total number of training iterations is set as 9000 epochs. The training process takes about 2 days on 4 Tesla V100 GPUs with the image batch size of 14. For training with pre- traitraining, pre-train the model for 50 epochs, then fine-tune the model on CTW1500 1550 and Texa by the same setting as training without pre-training states.
4.1 COMPARISON WITH THE STATE- STATE-OF-THE-ANE METHODS
To demonstrate the effectiveness of our method, we take (1) point-based methods ABCNet (Liu
et al. 2020), TextRay (Wang et al. 2020a). DRRG (Zhang et al. 2020). ContourNet (Wang et al.
2020ch and TextBPN (Zhang al. 2021; ; (2) segmentation- methods PSENet (Wang al.
2019a). PAN (Wang et al. 2019b). DE (Liao et al. 2020). FCENet (Zhu et al.] (021b), and
DB
(Liao al. 2022); and (3) recent transformer-based methods FewBetter (Tang et al. 2022)
7

---

Under review as a conference paper at ICLR 2023
(8)
(h)
641
19
TELSK Potvero Phooto 1001 A ACIINN
Hap ChISE
FORT LALDEROLL
AAAAE PHILLIES
Happy
FORT LAIRERILL
EXLS
PHILLIES
Happy
PHILLIES
Hapo
PHILLIES
Qualitative comparisons with previous SOTA on Texal-Text and CTW1500. Com- pared DB++ and FCENet, our PBFormer predicts texts (the first two are DB++'s Total-Text detections, and the last four are FCENet's CTW1500 de- tections, because they did not release the model of another dataset). Compared to TESTR, PBFormer reduces false negatives and performs better for long and curved texts.
Figure 4:
compact and precise contours for crowded
MUSOOD DRASS
Figure 5: Effect visualization of shape-constrained loss. For each image pair, the left image shows the results with the fitting loss Eq. 7 and the right image is with the shape-constrained loss Eq. [I
With the chape-constraine loss, PBFormer outputs more complete contours.
A'
A'
A A
A
A'
NB
NB
Figure 6: Visualization of CPA'S attention maps. The left four images show CPA produces attention for the small texts at swallow layer and the large texts at a deep layer. The right four image show attention is concentrated on one layer due to texts having similar sizes.
and TESTR Zhang et al. 2022b). TESTR* means we use their official training codes, settings, and models but only set the recognition loss zeros. For PBFormer, we will not use text character annotation for text detection training.
Models trained from scratch. As shown in Tab. 1 PBFormer establishes a new state-of-the-art of 87.0% on CTW1500, which is 1.7% better than previous best TESTR while achieving 4.4 x FPS. Moreover, PBFormer yields the best F-measure 87.1% on Total-Text, which is 0.2% better than previous best TextBPN while being 2.3 x FPS. Compared to TESTR, PBFormer also improves it by 0.8% while keeping 4.5 FPS.
Models with pre-training. Since previous methods have different choices of pre-training datasets, we choose to use the CurvedSynthText as FewBetter did. As shown in Tab. 2 PBFormer can achieve better results from the model pre-training. On the CTW1500 dataset, our method achieves the best results. It outperforms previous best TESTR by 0.9% in terms of F-measure and is 4.4 x faster than TESTR. On the Total-Text dataset, PBFormer also has the best performance. Compared to TESTR, PBFormer performs 4.5 x FPS while being a slight 0.1% higher F-measure. Without training TESTR's recognition branch, PBFormer's improvements are more obvious by +1.4% and +0.7%.
8

---

Under review as a conference paper at ICLR 2023
In addition, PBFormer yields 2.7% and 2.1% F-measure better than the previous best segmentation- based DB++ on CTW 1500 and Tex,,-Text, while keeping a very close FPS performance.
Qualitative comparisons. Considering crowded texts in Fig. a).(c), and (f), PBFormer performs fewer false-negatives than TESTR and more accurate contours than DB++ and FCENet. When texts have very long shapes or have characters' large scale-changes, PBFormer detected more completed contours than DB++, FCENet, and TESTR, as Fig. d() and (c) have shown.
Table 3: Comparison on Texal-Text of the same Table 4: CPA's influences and comparisons with network with different text representations. CPA
FPN and ASE. Ext. means trainable parameters.
Ref.
Rep.
F.
Prec.
Rec.
Module
Ext.
F.
Prec.
Rec.
PB
85.8
90.2
81,9
86.0
90.5
81.9
Pts
82.8
88.4
77.8
Enlarge
85.2
88.9
81.8
Bez
83.9
89.8
78.7
Attn.
86.1
90.6
82.1
PB
86.0
90.5
81.9
CPA
87.1
92.1
82.6
Pts
83.2
87.6
79.1
ASF
85.3
90.9
80.4
Bez
84.2
89.7
79.4
FPN
85.0
89.5
81.0
PB
87.1
92.1
82.6
Pts
84.6
90.5
79.5
Bez
85.3
90.9
80.4
4.2 ABLATION STUDY
Investigation of proposed components. In this section, investigate the effects of different text representations under the same network and validate the effects CPA and refined reference points. Tab, 3 shows the following conclusions: (I) PB outperforms polygon points and Bezier con- trol points in every network configuration. In Fig. 12 we also show that PB can distinguish indi- vidual texts facing adjacent and overlapping texts better than other representations; (2) the improve- ments of CPA and refined reference points are consistent for different text representations.
To further investigate the effect of the CPA module, Tab. shows the performance of the models with different CPA configurations and the models replacing CPA with other fusion modules. The study of the CPA configuration has the following conclusions: (1) just enlarging features performs 0.8% worse; (2) only using attention has a minor 0.1% improvement; (3) combining both boosts the performance by 1.1% significantly. The effectiveness of the CPA module is that the attentional fusion adaptively highlights texts' features at suitable scale and suppresses the features of other scales. 6 demonstraats the four attention maps across scales from shallow to deep layers of the backbone. Moreover, CPA performs 1.8% and 2.14 1% better than FPN and ASF (Liao et al.) 2022) We attribute to (1) FPN (Lin et al.) 2017) will NOT improve the object detection performance because the cross-level feature exchange already adopted by the multi-scale deformable attention module (Zhu et al., 2021a). FPN degrades text detection performance because introduced additional parameters are not learned well because the text dataset we used is much smaller than the common object dataset. (2) CPA's selective mechanism is more compatible with DETR-like detectors than ASF. The latter works better with DB++, which tends to preserve both the global structures and local details for accurate segmentation masks. A detailed comparison can be found at Sec. 6.2]
Effect of shape-constrained loss. Using the shape-constrained loss, the performance on Total- Text increases from 83.5% to 87.1%. %. As Fig. shows, the model trained with shape-constrained loss can produce more complete contours (the right image in the pair) than the models without the loss (the left image in the pair).
5 CONCLUSION
We have presented PBFormer, an efficient and accurate text detection method. It is superior to han- dle crowded texts or texts with diverse shapes. PBFormer equips a new text representation, Polyno- network consisting of a cross-scale pixel attention module and
mial Band, to transformer-bace lightweight deformable transformer. We supervise the network with a encouraging the network to output the correct contour length. PBFormer shows strong robustness when training without pre-trane tre-training on the additional datasets, which is much more resource- ffiinddy than other transformer-base methods.
loss term,

---

Under review as a conference paper at ICLR 2023
REFERENCES
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and
Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, pp. 213-229, 2020.
Chee Kheng Chng and Chee Seng Chan. Total-text: comprehensive dataset for scene text detection and recognition. In ICDAR, pp. 935-942. IEEE, 2017.
Chee Kheng Chng, Errui Ding, Jinguuo Liu, Dimosthenis Karatzas, Chee Seng Chan, Lianwen Jin, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, and Junyu Han. ICDAR2019 robust reading challenge on arbitrary-shaper text rre-art. In ICDAR, pp. 1571-1576. IEEE, 2019.
Dan Deng. Haifeng Liu, Xuelong Li, and Deng Cai. Pixellink: Detecting scene text via instance segmentation. In AAAI, pp. 6773-6780 AAAI Press, 2018.
Zhanghui Kuang, Hongbin Sun, Zhizhong Li, Xiaoyu Yue, Tsui Hin Lin, Jianyong Chen, Huaqiang Wei, Yiqin Zhu, Tong Gao, Wenwei Zhang, Kai Chen, Wayne Zhang, and Dahua Lin. Mmoer: A comprehensive toolbox for text detection, recognition and understanding. arXiv preprint arXiv:2108.06543 2021.
Hui Li, Peng Wang, and Chunhua Shen. Towards end-to-end text spotting with convolutional recur- rent neural networks. In ICCV, pp. 5248-5256. IEEE Computer Society, 2017.
Hui Li, Peng Wang, and Chunhua Shen. Towards end-to-end text spotting in natural scenes. CoRR, abs/1906.06013. 2019.
Minghui Liao, Baoguang Shi, and Xiang Bai. Textboxes++: A single-shot oriented scene text detector. IEEE Trans. Image Process., (000)3666-3990 2018a.
Minghui Liao, Zhen Zhu, Baoguang Shi, Gui-Song Xia, and Xiang Bai. Rotation-sensitiv regres- sion for oriented scene text detection. In CVPR, pp. 5909-5918. Computer Vision Foundation / IEEE Computer Society, 2018b.
Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai. Real-time scene text detection with differentiable binarization. In AAAI, pp. 11474-11481. AAAI Press, 2020.
Minghui Liao, Pengyuan Lyu, Minghang He, Cong Yao, Wenhao Wu, and Xiang Bai. Mask textspot- ter: An end-to-end trainable neural network for spotting text with arbitrary shapes. IEEE Trans. Pattern Anal. Mach. Intell., 43(2))532-544 2021.
Minghui Liao, Zhisheng Zou, Zhaoyi Wan, Cong Yao, and Xiang Bai. Real-time scene text detection with differentiable binarization and adaptive scale fusion. CoRR, abs/2202.10304 2022.
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Be- longie. Feature pyramid networks for object detection. In CVPR, pp. 936-944. IEEE Computer Society, 2017.
Ruijin Liu, Zejian Yuan, Tie Liu, and Zhiliang Xiong. End-to-end lane shape prediction with trans- formers. In WACV, pp. 3693-3701, 2021.
Ruijin Liu, Dapeng Chen, Tie Liu, Zhiliang Xiong, and Zejian Yuan. Learning to predict lane shape and camera pose from single image via geometry constraints. In AAAI, pp. 1765-1772. AAAI Press, 2022.
Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang. Curved scene text detec- tion via transverse and longitudinal sequence connection. Pattern Recognit. 90:337-345, 2019.
Yuliang Liu, Hao Chen, Chunhua Shen, Tong He, Lianwen Jin, and Liangwei Wang. Abenet: Real- time scene text spotting with adaptive bezier-curve network. In CVPR, pp. 9816-9815. Computer Vision Foundation IEEE, 2020.
Shangbang Long, Xin He, and Cong Yao. Scene text detection and recognition: The deep learning era. Int. J. Comput. Vis., 129(1):161-184, 2021.
10

---

Under review as a conference paper at ICLR 2023
Chixiang Ma, Lei Sun, Zhuoyao Zhong, and Qiang Huo. Relatext: Exploiting visual relationships for urbitrary-shaped scene text detection with graph convolutional networks. Pattern Recognit. 111:107684, 2021.
Nibal Nayef, Cheng- Lin Liu, Jean-Mare Ogier, Yash Patel, Michal Busta, Pinaki Nath Chowdhury, Dimosthenis Karatzas, Wafa Khlif, Jiri Matas, Umapada Pal, and Sean-Chrittophe Burie. IC- DAR2019 robust reading challenge on multi-lingual scene text detection and recognition RRC- MLT-2019. ICDAR, pp. 1582-1587. IEEE, 2019.
Xugong Qin, Yu Zhou, Youhui Guo, Dayan Wu. Zhihong Tian, Ning Jiang, Hongbin Wang, and Weiping Wang. Mask all you need: Rethinking mask R-CNN for dense and arbitrary-shaped scene text detection. In ACM Multimedia, pp. 414 423. ACM, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Pruceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021.
Sangeeth Reddy, Minesh Mathew, Lluís Gómez, Marçal Rusiñol, Dimosthenis Karatzas, and C. V. Jawahar. Roadtext-Ik: Text detection & recognition dataset for driving videos. In ICRA, pp- 11074-1 1080. IEFE, 2020.
Tao Sheng, Jie Chen, and Zhouhui Lian. Centripetaltext: An efficient text instance representation for scene text detection. In NeurlPS, pp. 335-346, 2021.
Baoguang Shi, Xiang Bai, and Serge J. Belongie. Detecting oriented text in natural images by linking segments. In CVPR, pp. 3482-3490. IEEE Computer Society, 2017.
Yipeng Sun, Chengquan Zhang, Zumingg Huang, Jiaming Liu, Junyu Han, and Errui Ding. Textnet: Irregular text reading from images with an end-to-end trainable network. ACCV pp. 83-99. Springer, 2018
Jingqun Tang, Wenqing Zhang, Hongye Liu, Mingkun Yang, Bo Jiang, Guanglong Hu, and Xiang Bai. Few could be better than all: Feature sampling and grouping for scene text detection. In CVPR. Computer Vision Foundation / IEEE, 2022.
Jun Tang, Zhibo Yang, Yongpan Wang, Qi Zheng, Yongchao Xu, and Xiang Bai. Seglink++: De- tecting dense and arbitrary-shaped scene text by instance-aware component grouping. Pattern Recognit., 96, 2019.
Zhuotao Tian, Michelle Shu, Pengyuan Lyu, Ruiyu Li, Chao Zhou, Xiaoyong Shen, and Jiaya Jia. Learning shape-aware embedding for scene text detection. In CVPR, pp. 4234-4243 Computer Vision Foundation IEEE, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 5998-6008, 2017.
Fangfang Wang, Yifeng Chen, Fei Wu, and Xi Li. Textray: Contour-based geometric modeling for sebitrary-shaped scene text detection. In ACM Multimedia, pp. 111 111- ACM, 2020a.
Hao Wang, Pu Lu, Hui Zhang, Mingkun Yang, Xiang Bai, Yongchao Xu, Mengchao He, Yongpan Wang, and Wenyu Liu. All you need is boundary: Toward arbitrary-shaped text spotting. In AAAI, pp. 12160-12167. AAAI Press, 2020b.
Wenhai Wang. Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Gang Yu, and Shuai Shao. Shape robust text detection with progressive scale expansion network. In CVPR, pp. 9336-9345 Computer Vision Foundation IEEE, 2019a.
Wenhai Wang, Enze Xie, Xiaoge Song. Yuhang Zang, Wenjia Wang, Tong Lu, Gang Yu, and Chun- hua Shen. Efficient and accurate urbitrary-shaped text detection with pixel aggregation network. In ICCV, pp. 8439-8448. IEEE, 2019b.
11

---

Under review as a conference paper at ICLR 2023
Yuxin Wang, Hongtao Xie, Zheng-Jun Zha, Mengting Xing, Zilong Fu, and Yongdong Zhang. Con- tournet: Taking a further step toward accurate urbitrary-shaped scene text detection. In CVPR, pp- 11750-11759 Computer Vision Foundation / IEEE, 2020c.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, pp. 5987-5995, IEEE Computer Society, 2017.
Fangneng Zhan and Shijian Lu. ESIR: end-to-end scene text recognition via iterative image rectifi- cation. In CVPR, pp. 2059-2068. Computer Vision Foundation / IEEE, 2019.
Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R. Manmatha, Mu Li, and Alexander Smola. Resnest: Split-attention networks. In CVPR Workshops, pp. 1735-2745 IEEE, 2022a.
Shi-Xue Zhang, Xiaobin Zhu, Jie-Bo Hou, Chang Liu, Chun Yang. Hongfa Wang, and Xu-Cheng Yin. Deep relational reasoning graph network for arbitrary shape text detection. In CVPR, pp. 9696-9705, Computer Vision Foundation IEEE, 2020.
Shi-Xue Zhang, Xiaobin Zhu, Chun Yang, Hongfa Wang, and Xu-Cheng Yin. Adaptive boundary proposal network for arbitrary shape text detection. In ICCV, pp. 1285-1294. IEEE, 2021.
Xiang Zhang, Yongwen Su, Subarna Tripathi, and Zhuowen Tu. Text spotting transformers. In CVPR. Computer Vision Foundation / IEFE, 2022b.
Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. EAST: an efficient and accurate scene detector. CVPR, pp. 2642 2651. IEEE Computer Society, 2017.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In ICLR. OpenReview.net, 2021a.
Yiqin Zhu, Jianyong Chen, Lingyu Liang, Zhanghui Kuang, Lianwen Jin, and Wayne Zhang. Fourier contour embedding for arbitrary-shaped text detection. In CVPR, pp. 3123 3131. Computer Vi- sion Foundation / IEEE, 2021b.
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
Sec. 6.1
states the differences between polynomial curve and Bezier curve.
Sec. 6.
states the differences between CPA and ASF (Liao al. 2022).
Sec. 6.3
states the architecture differences compared with TESTR (Zhang et al. 2022b) and Few-
Better
aangen al. 2022).
shows how we generate ground truth point sets for top, bottom, left and right sides from original annotated polygon points.
Sec. 6.4
Sec. 6.5
conducts experiments on DASTI500 (Tang et al. 2019), MLT2019 (Nayef et al. 2019 (Chng al. 2019) and show some qualitative results on them.
and Ar T019
Sec. 6.6
visualizes comparisons different text representations with the same network.
Sec. 6.7
investigate learning configurations of polynomial bands.
Sec. 6.8
shows efficiency's comparisons of different text representations.
Sec. 6.9
shows visualizations attention maps.
Sec. 6.
shows some failure cases.
Sec. 6.11
shows more qualitative results CTW1500 and Texxl-Text.
6.1
DIFFERENCE BETWEEN POLYNOMIAL CURVE AND BEZIER CURVE
The polynomial curve is another form of the fitting function, but better form.
Let us consider the parameters optimized in our loss functions. For the polynomial curve, the opti- mized parameters are for the function defined in the image space. For the Bezier curve, the optimized parameters, i.e., the control points, are the function defined with a specifie variable "(".
Important: The form of the curves will finally determine how optimize their parameters. (1) For the polynomial curve, we can measure the difference between the annotated points and the predicted polynomial function by comparing evenly sampled indicated polynomial curve and ground truth text-line. Such a metric can better reflect how humans percept the shape difference. (2) For the Bezier curve, need first fit the ground truth obtain the control points by least square method, then supervise the predicted Bezier curve by comparing the difference between the control points. Notably, the mismatch in Bezier's control point cannot reflect the difference the curve shape. A slight difference in the control points may bring large shape change.
Can be Bezier curve learned like a polynomial curve in the image space? The answer NO.
The Bezier curve is defined as:
c ctt) Σ byB (t),0 1=
(13)
are (1- (
where {by = (I,, (z)),0))} are Bezier control points, and B3n (t) =
Bernstein basis polynomials. Please note that the Bezier curve e (1) is polynomials of / rather than I, with the value for being fixed between 0 and 1. If (t) samples K points by taking O, K 2 1, the sampled points in the image space are not evenly arranged. Therefore, ctt)
cannot calculate loss with text-line in the image space.
For a clear understand, we give a simple Python code and visualization to show the above issue.
#1 !!----- coding: utf-8 import numpy as np
import matplotlib.pyple
as plt
from scipy. interp.itte import interpld Define Bezier control points
PO, P1, p2 np.array 01, [2, 4], [5, 311)
13

---

Under review as a conference paper at ICLR 2023
4.0
3.5
3.0
2.5
2.0
1.0
0.5
0.0
3
...
Curve Sogment in Image Space
Bezier Control Point of the Curve Segment
Evenly Sumpled by c(t) in Space Evenly Sampled in coordinate (image space)
Figure 7: The visualization of the mismatch about two point sequence. Given curve segment and its corresponding Bezier curve (represented by Bezier control points, red pentagram), points evenly sampled in image space are not matched with points evenly sampled in space of Bezier curve (t) in Eq. 13
plt. www.sscatteeeee (P)[0], PO PO[1], a- red', s-300, ma marker=(x()
pit. scatter (P110), PI[l|, o= red', s=300, marker=/ k() pit. scatter (P2101, P211], red', s=300, marker='*")
11 Define Bezier curve
12 P lambda t: (1 - +))+2 + PO 2 x t (1 t) P1 t**2 P2
13 Equally sample 11 points in space
points np. array
([P ([P for t in np. linsinspace (0, 1, 11)1)
13 x, y pointsl:, 0], points[:, 1]
16 plt.plot (x, y, color=/black', linestyle='--))
17 scattert (x, y, c='greent
18 Equally sample 11 points in space
fx = interpld y, fill "ul_valle==xxxtapolatt
20 _ np.linspace (0, 5, 11)
21 y - =_fxx(x)
22 plt. scaater show()
g- orange""
Listing 1: Python code to show evenly sampled points Bezier's space cannot be matched with evenly sampled points in image space.
As Fig. 7 shows, Given curve segment (dashed line), with equally distance =055. Then, given a Bezier curve which represents the curve segment, we can also sample 11 points from t=0 to t=1 with equally distance 1,= As points evenly in space (green circles) are not matched with evenly sampled points in image space (orange circles). Thus Bezier curve cannot be learned like polynomial curve the image space.
sample 11 points from ==0 to 2=5
see, sampled
6.2 DIFFERENCES BETWEEN CPA AND ASF.
Although CPA and ASP Liao et al. 2022 utilize multi-scale feature maps, they are very different in motivation and implementation.
14

---

Under review as a conference paper at ICLR 2023
Motivation difference:
ASF works with DB++, a segmentation- method to predict the pixel-wise text region. The motivation of ASF to help DB++ obtaín a more accurate segment mask, which includes both global structures and local details.
CPA works with PBFormer, a detection-based method to predict the parameters of object shape. It known that the detection results sensitive the scale, and the motivation of CPA is to highlight the best scale and suppress the others. Our experiments demonstrate that the selection mechanism of CPA is particularly crucial for DETR-like detectors (including our method) in text detection, which don't 't have non-maximum (UP)) for post- processing. If we replace CPA with ASF, as in Table the performance will drop 1.8% in terms of the F-score.
Implementation difference:
ASF performs spatial attention. first generates the mask weights for different feature maps by stack of Conv-ReLU and Conv-Sigmoid modules, then concatenates the mask- weighted feature maps together.
CPA performs scale attention. aligns the feature maps of the different scales by enlarging the small-scale feature maps. The attention map is not obtained by introducing a new network module but by comparing the values the existing feature maps among different scales with SoftMax. All the feature maps weighted with the attention map the resized to the original size.
Explanation summary:
To preserve both the global structure and local details of the segmentation mask, ASF performs spatial attention for each feature map and concatenates the information of all the feature maps.
To produce the detection results with accurate scale, CPA performs scale attention among different feature maps and selectively highlights the feature maps of the best scale and suppresses the feature maps of the other scale.
We assume the selective mechanism of CPA is compatible with DETR-like detectors that not have non-maximum suppression. Two pieces of evidence: if we replace CPA with ASF, the performance drop; We only utilize two-layer transformer encoders and decoders, much less than the TESTR, indicating the effectiveness of the fused feature map.
6.3 RELATIONS TO TESTR AND FEWBETTER
PBFormer has multiple efficient and useful designs compared with TESTR (Zhang et al. 2022b) and FewBetter (Tang et al. 2022).
Compared with TESTR:
PBFormer is single-stage without relying on intermediate bounding box results. In contrast, TESTR adopts the two-stage Deformable-DETR. predicts the bounding box in each fea- ture point of the transformer encoder's output, then selects topK boxes based on confidence embed them into positional embeddings and reference points, as Fig. (a)'s red block shows.
PBFormer adopts a coarse-to-fine strategy to generate reference points for deformable at- tention modules. The first decoder's output encodes 2-d residuals to move the reference point into a better localization. In contrast, TESTR uses the same reference points for all deformable attention modules. Fig. g (a) and (c)'s orange arrows illustrate the differences.
PBFormer inserts Cross-scale Pixel Attention module, which selectively highlight text regions at suitable scales. The selective mechanism CPA compatible with DETR- -ike detectors that do not have non-maximum suppression, enabling PBFormer to only utilize two layers of transformer encoders and decoders. In comparison, TESTR uses six layers of transformer encoders and decoders.
15

---

Under review as a conference paper at ICLR 2023
Regrosii
So Raueppeet Decider Leym
(a) TESTR
(b) FewBetter
(c) PBFormer
Figure 8: Detailed Network Comparisons with TESTR and FewBetter. (a) NumEnc denotes the sequence length of encoder's output, which equals the total spatial quantity across multi-scale feature maps. (c) The proposed CPA module has not any trainable parameters.
PBFormer is only pre-trained CurvedSynthText, while TESTR is pre-trained on com- bination of three datasets, i.e., CurvedSynth MLT2017, and Total-Text.
Compared with FewBetter:
PBFormer does not need FPN in the backbone or generate segmentation maps. Differently, FewBetter has an FPN network to generate convincing segmentation masks for feature selections, as Fig. (b)'s ' blue block shows.
PBformer utilizes deformable transformer while FewBetter the original transformer. Besides, we also use two transformer encoder and decoder layers, while Few Better uses the six for encoder and decoder, respectively.
6.4 GROUND TRUTH OF ToP, BOTTOM. LEFT AND RIGHT GENERATION.
In this section, we introduce how to generate four ground truth point-sets for the top, bottom, left and right sides.
In common text datasets, a text instance is annotated by 2K discrete points. For example, K = 5 in Total-Text and K 7 in CTWI500 More importantly, these points annotated along with human reading hobbies. They are ordered in counterclockwise order, and the first point is always the top- left of the first character. The former K points PI....PA and the latter K points PK+1. .Px are located as Fig. 9 shows (green for the former K, blue for the latter K).
Therefore, the generating process of ground truths for the top, bottom, left, and right curve follows subsequent five steps:
1. Divide 2K annotations into four Sets. S {p..... .p.....pp,, S .p.+ .Pak
8 = [Px Px+1), and S = [P2K P1}.
2. Assuming S1 and 2 would fitted by a mapping / : - y. judge whether assumed functions are both single-valued. If yes, go to step 3. If not, go step 4.
3. S' and 2 both use the form y = (a); S² and S both use the form x f (3). If Ss middle point is higher than S2's, S1 becomes the ground truth for the top curve, and 2
16

---

Under review as a conference paper at ICLR 2023
THE NORTH BRITISH ESSTEEER NB
single-valued?
St is fitted by f(y)
S' right
EST
COMPANY LTD
companion
St single-valued?
St fitted by (ty
is left
1885
single-valued?
n0
S fitted by f(x)
S is as
Cenner's y
S single-valued?
fitted by (())
S" bottom
vanglov
single-valued?
singlo-valued?
single-valued!
yes
yes
is fitted (x)
is finted
The first point of originally unnotated polygon points
fitted by y
St fined by ((y)
The tirst hall' of originally annotated polygon points
Conte's tipprroon
Conter's compaction
S' is at top
S is at bottom
St is left
Stis is
second balf originally anmotated polygon points
Figure 9: Visualization of generating ground truth for the top, bottom, left, and right sides from raw annotations. The black rounded green dots represent the first annotated point for text.
becomes the ground truth for the bottom curve, and vice versa. If middle point is more left than $4 S becomes the ground truth for the left curve, and S becomes the ground truth for the right curve, and vice versa. Go to step 5.
4. S and 22 both middle point is more left than S s, S becomes the ground truth for the left curve, and $ becomes the ground truth for the right curve, and vice versa. If SH middle point is higher than 's 8 becomes the ground truth for the top curve, and 4 becomes the ground truth for the bottom curve, and vice versa. Go to step 5.
the form
f (y): S and S both use the form y y (x). If St's
5. Sample S1, S $3 and S evenly to generate dense supervision points (for PB's loss calculation, see Sec. 3.3
Single-valued condition. A mapping f + which is used to fit a set of order points satisfies the single-valued condition if and only if:
T1 < 22
xn x1 x2
In
(14)
where 21, EI....,
Cr is the x-coordinate sequence of order points, and n denotes the number points.
As Fig. 9 shows, illustrate the processes for horizontal reading direction text "COMPANY" and a vertical reading direction text "DISTILLERY."
Table 5: Comparisons on DASTI5000 testing set. Rec.
Table 6: Experiments MLT2019 and Art2019.
Method
Prec.
Method
Rep. F.LT
F.Art
TextBoxes
50.9
67.3
40.9
DB++
Seg
64.7
71.1
RRD
53.0
67.2
43.8
FCENet
Seg
66.3
74.0
EAST
62.0
70.0
55.7
TESTR
Pts
68.7
75.2
SegLink
65.3
66.0
TESTR
Bez
66.7
73.6
CTD+TLOC
66.6
73.8
60.8
PBFormer(ours)
PB
69.3
76.2
PixelLink
74.7
74.5
75.0
ICG
79.4
79.6
79.2
ReLaText
85.8
89.0
82.9
MAYOR
86.6
85.5
PBFormer(ours)
85.9
90.0
82.1
PB Fotee(((woo-stgge)
86.6
90.2
83.2
17

---

Under review as a conference paper at ICLR 2023
DAST1500
Ar12019
APP KRIVES
LEVEE
CRICETS TO
MLT2019
OTIS
ARTISAN PEI Peinture Decoration
SURJUMAN
MORAL ERIC
STOCKYARDS STATION
Ltt K HUTK STUPHIA REGEN
THEATRE DUS CORPS SHINT
DOULDNBERIE ELBARBKA
Rosé terroiy
it Sotha
Lltta wall
ELBARAKÁ e المركة
Figure 10: Qualitative results on DASTI500, ArT2019, and MLT2019.
6.5 EXPERIMENTS ON DASTI500, MLT2019, AND ART2019
We follow MAYOR Qin et al. 2021 to conduct experiments on DASTI500 (Tng et al. 2019 and compare with TextBoxes (Lio al.
2018a), RRD (Liao al. 2018b). EAST (hoou et al. 2017]
SegLink (Shi et al. 2017). CTD+TLOC (Liu et al. 2019). Link (Deng et al. 2018) ICG Tang
ct al.
2019), ReLaTlet (Ma et al. 2021), MAYOR (Qnn et 2021).
Methods on leaderboards of ICDAR MLT 2019 (Nayef et al. 2019) and ICDAR ArT 2019 Chnng
al. (2119) adopt ResNeSt-101 (Zhang et al. (002a)), ResNeXt-lor (Xie et al., 2017). or ResNext- 152 (Xie et al. (017) their backbones which are far heavy for a fair comparison. Therefore, we conduct experiments across DB++ (Liao et al., 2022). FCENet (Zhu et al. 20216), TESTR (Zhang al.) 2022b), and PBFormer by splitting the training set of MLT2019 and 1rt2019 into training and validation with 1:1. We generate ground truth of Bezier control points following ABCNet (Liu al. 2020). We fine-tune all models' best checkpoints the training split 1000 epoch.
Results are shown in Tab. 5 and Tab. 6 PBFormer achieves comparable results with previous state- of-the-art DASTI500. We attribute to (1) DAST1500 are full of dense texts. MAYOR is two- stage and has spatial designs for those cases, such as region proposal network (RPN) and MLP Mask Decoder (MLM). (2) PBformer is single-stage. Although it performs less well than MAYOR, it is
18

---

Under review as a conference paper at ICLR 2023
more efficient because it does not have post-processing with NMS. If take training strategy same with two-stage Sermmalle-ETR., we can achieve better performance (86.7 vs. 86.6). Besides, one-stage PBFormer still outperforms MAYOR CTW1500 (87.0 vs. 85.3) and Total-Text (87.1 vs. 86.3).
Moreover, PBFormer outperforms DB++, FCENet, and TESTR MLT 2019 and Art 2019. We also show some visualizations on DASTI500, MLT2019, and ArT2019 in Fig. 10
6.6 VISUALIZED COMPARISONS OF DIFFERENT TEXT REPRESENTATIONS WITH THE SAME NETWORK.
In this section, we adopt the same network architecture, PBFormer, but change the output head using Bezier control points, polygon points, and polynomial bands. In Fig. 12 we show comparisons of adjacent and overlapping texts. For adjacent texts, PB can distinguish individual texts correctly, but the others are likely to combine them falsely. For overlapping texts, PB can detect complete contours, while others are likely to detect incomplete contours or lose the overlapped texts.
6.7 INVESTIGATION OF PB's CONFIURRATIONS.
Table 7: Comparisons of the number of different supervision points per polynomial curve.
Table 8: Comparisons of polynomial's 's order.
Curve
F.
Prec.
Rec.
Number F.
Prec.
Rec.
Quadratic
87.1
92.1
82.6
6 84.6
90.5
79.5
Cubic
87.0
92.2
82.4
12 86.0
90.4
81.9
Quartic
86.7
91.0
82.7
24
87.1
92.1
82.6
30
86.5
91.1
82.4
36
86.4
91.1
82.3
We first analyze the influence of the polynomial order of PB. Theoretically, higher-order polyno- mials could fit more complex texts, but the quadric curves can describe the texts most current datasets well. For this reason, the higher-order curves overqualified represent the text shapes in the existing datasets, but they are easily overfitted, Secondly, we analyze the effect of sampling points' density. As Tab. shows, too few points perform 2.5% worse since the quantity is insuffi- cient to learn curved shapes, while too many points also decrease 0.7% due to redundant learning points, especially for straight texts.
6.8 EFFICIENCY'S ' COMPARISONS OF DIFFERENT TEXT REPRESENTATION
The polynomial band more efficient than segmentation masks and polygon points during infer- ence, Fig. I1 shows. The polynomial band is more efficient than Bezier points during ground truth generations.
sparse for evaluation, such as 16 points in TESTR, while more points are needed to compute accurate IoUs in MMOCR's evaluation protocol (Kuang al., 2021). especially for curved texts. Therefore, as Fig. 1 shows, after dividing points into splines, each of which is fitted by a polynomial, then evenly sampling. In contrast, PB avoids the above two-stage iterative procedure because PB direct contains four curves for the whole contour, which generates dense contour by sampling.
Polygon points need spline interpolation because the number of network's output
Ground truths of Bezier points are calculated by the least square algorithm based on original polyg- onal annotations (Liu et al., (2020). which consumes additional time and resources for generating ground truth for extra data. Differently, PB directly utilizes the polygonal annotations without addi- tional consumption to transform raw annotations.
6.9
MORE VISUALIZATIONS ON ATTENTION MAPS.
In Fig. 13 we select some representative attention maps of the cross-scale pixel attention module. The CPA capable of attending to text regions at suitable layer adaptively. We see CPA leamed to attend at a single layer if texts have similar sizes. When different texts have size various, CPA learned to attend to relatively small texts the swallow layer while large texts at the deep layer.
19

---

Under review as a conference paper at ICLR 2023
Network Output
Generation Process
Description for Eval
Segmentation
Grouping
Contour Detection
Text Contour
Polygon
Divide into Spline
Line Fitting
Sampling
Text Contour
Bezier Points
Sampling
Text Contour
Polynomial Band (ours)
Sampling
Text Contour
Comparisons of the process from network's output representation to final text con- tour. For segmentation maps, we only show main post-process cost-processing since there might be other prac- tical processes such as outlier removal, distance (or directional) maps refinement, etc.
Figure 11:
6.10 FAILURE CASES.
In Fig.
14 visualize failure cases in two situations: (1) texts which are extremely small as a. shows; (2) texts which occupy a minority in common datasets, such as chinese-lingual Fig. b) and texts processed into symbols beautified by art-style.
Fig.
texts
The limitations may inspire our future work. Firstly, enlarging the input's resolution can improve the detection of small texts, but further consumes computations. Secondly, collecting multi- lingual texts or adopting art-style augmentation on the common texts for training will improve the recall of the results.
6.11 MORE QUALITATIVE RESULTS.
we present more qualitative results CTW1500 and Total-Text. PBFormer predicts not only accurate contour for curved texts but also handles multi-oriented data success- fully. More importantly, when texts crowded, PBFormer also distinguishes them clearly. Results indicate that our PBFormer robust in various scenes.
In Fig. 1. and Fig. 16
20

---

Under review as a conference paper at ICLR 2023
Bezier Control Points
Polygon Points
Polynomial Band
STREET DREAMS MOTORSPORT LLC
STREET DREAMS ORTORTPON LLC
STREET DREAMS POTONSPORT LL
NUTRITIOUS & DELICIOUS
NUTRITIOUS & DELICIOUS
NUTRITIOUS & DELICIOUS HINEOS RRINES SORIN
MEAMI:
HIMPE
Hoalihy
Healibo
Hesliin
Haalth
Unire.: EAHIVEMENEE
Usiog
ON THE 6O
ON THE 60
ON THE 00
HUDYY
RUOYS
NUOYS
LARRES
BARBIE
Adjacent
STRIGNG COLT
WELCORE COLYEON EXT
WILKIONG ENETTER
BAR RESTAURANT
BAR RESTAURANT TRUCHA CRIOLLE CA PIOUE MACHO MILANEZI Y CARNE PEJERREY
BAR RESTAURANT
SUBAD TRUCHA CRIOLLA PIOUE MACHO POLLO KILANZZA Y CARNE PEJERREY
SOBRD
SUGAR
TRUCHA CRIOLLA
PIQUE MACHO DE POPLO MILANEZA Y CARNE PEJERREY
DEPOLO
DENVER F F FIREE
DENVER
DENVER F FIRE
FBI D
Overlapping
FIRE
THE BUSH BEAST SAFETY HEIGHT
THE BUSH BEAST SAFETY HEIGHT
THE BUSH BEAST SAFETY HEIGHT
122cm (4f)
122cm (411)
122cm (4ft)
NO T
NS TUUUU
Figure 12: Visualized comparisons of different text representations with the same network.
21

---

Under review as a conference paper at ICLR 2023
A
MOM''S
COORDSSS SODY SHOP
BODY
OOD SHOP
A
A=
A
A
RELOAD BONANZA
RELOAD ONANZA
RELOAD
RELOAD ONANZA
A
KEMAB
A
KEMA BIOHTBOE BIST
A
KEMAR
A
KENAR LIOHTHOUSE LISTRICT
LORE HOO SEE DISTRICT
LIGHTHOUS DISTRICT
A'
A=
A
A
SELAMAT DATANG ME MORI TANTUNG ASAM
SELAMAT DATANG
SELANAT DATA
SELAMAT DATANG
MEMOR
MEMOR
MEMORI
D TANTUNG ASAM
D TANIUING AS ML
D TANJUNG ASAM
REAVIRO
A'
A
A
Proecue
A
Little Elly
Little
Eily
Little Eliy
Cittle blyy
Figure 13: CPA's attention maps Total-Text. A A 3 and A represent the attention map weighting the multi-scale feature from network's 's swallow to deep layers.
AUTRY
A ENGLE L
BLUE MONTANA SKIES
ールル
(n)
(e)
Figure 14: Visualization of failure cases. The yellow dashed boxes denote the failed detections.
22

---

Under review as a conference paper at ICLR 2023
Curved Texts
STEANBE R.G.
HOTEL CALLEE
RIG #3 IS #1 GOAL
WWe Won't 't Go BaCH WE Will Fight Back wiw.gis hrr.orr
NAZARETH
FURDES STELLGENNIG
BARNSTORMERS RESTAURA
DRITLING COMPANY
NOTICE
SHnO N. HESPLRGHP-SRR
ABTO PROOPERROTONEE 16 RUU
TIGRADE SEEDS
FOTILER
ГA3-21
*
AMERICAN BUILT *
LIGHEE BOSTON
FISKE
RAWSON
HAJAHE TATAPCTAH
ARMS COMPANY
Crowded Texts
EXPRESIONN
Crewn
WILCOVMLLEE
ROUNDER BEIGINAT
CHEESE
OOFFEE TTANE CHRCANO SuNtHiar
Evil
FILI fe churse
ROD WITH
STILTON
MOON
MIDWAY
rappcs
JUMBO
BEST PRALINES
walieiieee 2010
HAMBURGERS
Multi-oriented Texts
Jos.
WRSTTEN
BARBEIR
a Pannty
Opened in I861
KIRING COa
The Lue HAAD Mev
YOT bot HOTONE
WIGS HAIPPIEC
TQUOOS
Phil's
BRRGINGAA WHNG
JARKET
ALABASTINE
Figure 15: More Qualitative results on CTW1500. We demonstrate different types of texts, such as curved, crowded, or multi-oriented.
23

---

Under review as a conference paper at ICLR 2023
Curved Texts
TEXAS EMBASSY LONDOW
Cnirardeifi
BOUCHUN HESTUR FRANCAIS
Restaurane
L&E Trench
CANTINA
SAALORSSER
INDIAN HARBOR KIS
Rootin srane HOYmakeer
WING STOP
FMR
SKINHE ASS
YACHT CLUB
OE LARRY-DUT
Crowded Texts
Wheat
VITABELLA
GOLDEN GATE BRIDGE MAIN SPAN
RIDGWELL WINDOWS CONSERVATORIES 0I
TURING 1912-195 1934
Coos breaker
IIGG OF CASE 100VT JSEN
VEC FOOL VIDE 50000 HIES DRNENN
Campusee Scienor weat barn here 1
creaa
VIRAAA buttin NEW PIRS
Craans
Raisi
Multi-oriented Texts
ME ORIGIMAN SLACED
CUCINA BAR
ALLEASO Co
MULT-AGTION
WINES
NON
The concept Frescheal
Posh REAL NOBURT
BORE STREET
WILLIE NELSON
Glass
Little Elly D LEBEE TOM Chal
Figure 16: More Qualitative results on Total-Text. We demonstrate different types of texts, such as curved, crowded, or multi-oriented.
24


Reviews:
[{"review_id": "fshwxQ5gnUS", "reviewer": "Reviewer_Tjeh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel text representation named Polynomial Band. Cross-scale attention is used to enhance the feature of text regions. The fitting loss function is optimized to adapt to scene text detection task. The results on two benchmarks show the effectiveness.", "summary_of_the_review": "The contributions are incremental. The analysis and explanations are not enough. It is not enough to meet the quality of ICLR.", "strength_and_weaknesses": "Strength\n1. The paper proposes Polynomial Band, which uses four polynomial curves to represent the text instance. \n2. The optimized fitting loss is showed to be effective.\n3. The method can achieve promising performance without per-training.\n\nWeaknesses\n1. The cross-scale attention is similar to Adaptive Scale Fusion in DB++. Compared with general attention operation, why can the parameter-free one highlight text regions? The author should show the differences and give more detailed explanations.\n2. The motivation for CPA is not clear. It works similar to an encoder block in transformers, which may explain the use of fewer encoder blocks.\n3. Compared with Bezier curve, polynomial curve is just another form of the fitting function. It seems to be more complicated in terms of the parameters and loss calculation cost. It is recommended to show the advantages more clearly.\n4. The visualizations only display the increase of precision or recall, which I prefer to contribute to the strong feature extraction network rather than polynomial curves. The author better shows the superiority in \"distinguishing adjacent or overlapping texts\".\n5. The training epoches are much larger than previous methods. The good performance may come from the stacking of computing resources.\n6. To show the robustness, it is recommended to test on more challenging datasets, such as ICDAR MLT 2019, ICDAR 2019 Art and DASTA1500.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Lc8vlLc6zk", "reviewer": "Reviewer_FmtQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new representation of texts named Polynomial Band (PB) for text detection. The shape-constrained loss designed for PB considers both shape and the length. The model uses ResNet50 as the backbone followed by Cross-scale Pixel Attention (CPA) and Lightweight Deformable Transformer. CPA is parameter-free, but it can effectively suppress useless regions and thus improves the final results. The lightweight deformable transformer helps to achieve high FPS values. The experimental results show the effectiveness of the approach.", "summary_of_the_review": "The results are good and the representation seems powerful and reasonable. Other proposals are also effective. Therefore, I see a good value in the paper for the scene text recognition community. However, there are several weaknesses found in the paper as mentioned above and I am not fully convinced that this paper should be accepted in the current form.", "strength_and_weaknesses": "Strengths:\n\nA new text representation, PB, is a concise and powerful representation for most texts observed in wild. The shape-constrained loss is well designed for the representation.\n\nCPA is parameter-free and computationally inexpensive, but it is shown that it is better than other methods.\n\nThe reported absolute numbers look strong.\n\nWeaknesses:\n\nIn section 3.2, it discusses a potential weakness of the representation that it cannot represent lines that have multiple values for single x. Probably, it is true that it is rare to observe such cases in wild, but it is a fundamental limitation of the approach.\n\nText detection is often followed by text recognition using the detected boxes. Typically, features corresponding to the bounding boxes are extracted and fed to the recognition model. The paper does not discuss how to extract features (or pixels) if a text is represented by PB.\n\nIt was not clear how a text is represented by four lines. I guess there is no guarantee that the beginning and ending points of each line match each other and there should be gaps among the lines. In that case, how is a complete region defined given four lines (and how is IoU computed)?\n\nIt was not clear if the proposed representation was good or the model was good. Ideally, other representations should be implemented with the same model and perform fair comparisons among them.\n\nThe results in Table 3 are interesting. Assuming \"-\" is the baseline without anything, adding something other than CPA did not seem to help much or even made it worse. I was not sure if it was reasonable. If there was a good explanation on this, it would be good to have a discussion on it.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9tywLYD69VK", "reviewer": "Reviewer_cjr7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a scene text detector, PBFormer, using the transformer with a “new” text instance representation called Polynomial Band (PB). The proposed PB is able to represent a text with a complex shape since it utilizes four polynomial curves to fit the text instance’s top, bottom, left, and right sides. The proposed PBFormer is basically a transformer model with the proposed PB text representation. Extensive experiments have been conducted to verify the effectiveness of the proposed PBFormer and its superiority to other existing approaches. ", "summary_of_the_review": "As analyzed in my above comments, there exist both clear strengths and weaknesses in this paper. To sum up, I think the quality of this paper is slightly below the bar of ICLR. I will be happy to read the author's responses and discuss with other reviewers to make my final decision.", "strength_and_weaknesses": "Strengths:\nThe proposed PB text representation utilizes four polynomial curves with a fixed number of parameters to fit a text instance, while polygon-points-based methods require using different numbers of points to describe different text regions. This facilitates the learning of parameters for the predicted outline of a text region, and thus reducing the computational cost of the adopted transformer model. Extensive experiments have been conducted and the results are convincing. \n\nWeaknesses:\n1) The ABCnet [Liu et al. 2020] also adopted a fixed number of parameters to fit the outline of a text region. Therefore, what is the difference between the text representation based on Bezier curves [Liu et al. 2020] and the proposed PB base on Polynomial curves? No discussion regarding this issue has been provided. To this end, the novelty is relatively limited. The authors need to explain more at this point instead of simply claiming the PB’s advantages against polygon-points-based methods.\n2) Vey few technical contributions have been made in the whole network architecture. The three modules contained in the network all utilize well-known techniques. This leads to my concerns on the insufficient academic contribution of this paper. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "251pAX0Yauj", "reviewer": "Reviewer_Lb4V", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new text shape representation for scene text based on the coefficients of a quadratic polynomial fitting the four contours (upper, lower, left, right) of the text instance. Then, a transformer-based detector is trained to predict the coefficients of the polynomials for all text instances in an image. The transformer is designed to be lightwieght in order to improve efficiency. Experiments using standard datasets containing irregular texts show state-of-the-art results in detection accuracy and a very good efficiency in terrms of detection time.", "summary_of_the_review": "The paper proposes a novel representation for text representation and a transformer-based architecture to predict text shapes. Although results show SoA performance, it is not clear wether the good results come from the new representation. In addition, there are some details in the comparison with the SoA that should be better presented and discussed. ", "strength_and_weaknesses": "Strengths\n- The paper proposes a new compact parametric representation for text shape based on polynomials. It also defines the metric to match the points generated by the polynomial with the points in the ground-truth.\n- It defines a lightweigth transformer architecture with only 2 layers with a specific coarse-to-fine strategy to generate reference points at each transformer layer.\n-  A cross-pixel-attention module with no learnable parameters is introduced to perform pixel-wise attention at different feature scalses.\n- The method obtains state-of-the-art results in text detection on standard datasets, with a very good efficiency in terms of computation time.\n\nWeaknesses\n- It is not clear the contribution of the new polynomial band representation. In comparison with other similar methods (for instance TESTR), it is not clear whether the improvement in the results come from the new representation or from the slight variations in the feature encoding and transformer architecture. I miss some experiment using the same architecture to different shape representations (for instance, polynomials, poitns and bezier). It is important to clarify this point as this is the main contribution of the paper. Apart from it, the architecture proposed is very similar to the one proposed in TESTR.\n- In the experiments there are some results that should be better analyzed and discussed. Results for TESTR are provided without using the recognition branch in training which leads to lower results to those reported in the original TESTR paper. However, reported FPS for TESTR method in tables 1 and 2 are the values reported in the original paper using the dual branch for detection and recognition. I guess that, if only detection branch is used, FPS should be higher. For a fair comparison, FPS using only the detection branch should be reported in this case. \nFurthermore, even if the method proposed here is a detection-only method, I think that a fair comparison should include results of TESTR obtained training the whole pipeline (detection and recognition). Even if the goal is only detection, It could be valid to train TESTR using the two branches and use only the detection branch to obtain the results. \n- In the main paper, it is not clear in section 3.3 how the polynomial representation is matched to a ground-truth polygonal representation consisting of several points not necessarily equally distributed. In the appendix, there are some more detailes, but I think this should be made more clear in the main paper. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject