CentripetalText: An Efficient Text Instance Representation for Scene Text Detection

1 Introduction . In the past decade , scene text detection has attracted increasing interests in the computer vision community , as localizing the region of each text instance in natural images with high accuracy is an essential prerequisite for many practical applications such as blind navigation , scene understanding , and text retrieval . With the rapid development of object detection [ 29 , 19 , 7 , 18 ] and segmentation [ 22 , 46 , 26 , 50 ] , many promising methods [ 51 , 23 , 37 , 16 , 38 , 39 ] have been proposed to solve the problem . However , scene text detection is still a challenging task due to the variety of text curvatures , orientations , and aspect ratios . How to represent text instances in real imagery is one of the major challenges for scene text detection , and usually there are two strategies to solve the problem arising from this challenge . The first is to treat text instances as a specific kind of object and use rotated rectangles or quadrangles for ⇤Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . description . This kind of methods are typically inherited from generic object detection and often utilize manually designed anchors for better regression . Obviously , this solution ignores the geometric traits of irregular texts , which may introduce considerable background noises , and furthermore , it is difficult to formulate appropriate anchors to fit the texts of various shapes . The other strategy is to decompose text instances into several conceptual or physical components , and reconstruct the polygonal contours through a series of indispensable post-processing steps . For example , PAN [ 38 ] follows the idea of clustering and aggregates text pixels according to the distances between their embeddings . In TextSnake [ 23 ] , text instances are represented by text center lines and ordered disks . Consequently , these methods are more flexible and more general than the previous ones in modeling . Nevertheless , most of them suffer from slow inference speed , due to complicated postprocessing steps , essentially caused by this kind of tedious multi-component-based representation strategy . For another , their component prediction is modeled as a simple Dirac delta distribution , which strictly requires numerical outputs to reach the exact positions and thus weakens the ability to tolerate mistakes . The wrong component prediction will propagate errors to heuristic post-processing procedures , making the rebuilt text contours inaccurate . Based on the above observations , we can find out that the implementation of a fast and accurate scene text detector heavily depends on a simple but effective text instance representation with a robust post-processing algorithm , which can tolerate ambiguity and uncertainty . CT ( ours ) PAN ( 2019 ) DB ( 2020 ) 80 81 82 83 84 85 86 Fm ea su re ( % ) 50 75 10025 DB-ResNet18 DB-ResNet50 PAN-320 CT-320 CT-512 CT-640 PAN-512 PAN-640 Speed ( FPS ) pixels that can be shifted into the region of the same text kernel form a text instance . In this manner , we can reconstruct the final text instances fast and easily through marginal matrix operations and several calls to functions of the OpenCV library . Moreover , we develop an enhanced regression loss , namely the Relaxed L1 Loss , mainly for dense centripetal shift regression , which further improves the detection precision . Benefiting from the new loss , our method is robust to the prediction errors of centripetal shifts because the centripetal shifts which can guide pixels to the region of the right text kernel are all regarded as positive . Besides , CT can be fused with CNN-based text detectors or spotters in a plug-and-play manner . We replace SPN in Mask TextSpotter v3 [ 13 ] with our CentripetalText Proposal Network ( CPN ) , a proposal generation module based on CT , which produces more accurate proposals and improves the end-to-end text recognition performance further . To evaluate the effectiveness of the proposed CT and Relaxed L1 Loss , we adopt the design of network architecture in PAN [ 38 ] and train a powerful end-to-end scene text detector by replacing its text instance representation and loss function with ours . We conduct extensive experiments on the commonly used scene text benchmarks including Total-Text [ 2 ] , CTW1500 [ 47 ] , and MSRATD500 [ 45 ] , demonstrating that our method achieves superior or competitive performance compared to the state of the art , e.g. , F-measure of 86.3 % at 40.0 FPS on Total-Text , F-measure of 86.1 % at 34.8 FPS on MSRA-TD500 , etc . For the task of end-to-end text recognition , equipped with CPN , the F-measure value of Mask TextSpotter v3 can further be boosted to 71.9 % and 79.5 % without and with lexicon , respectively . Major contributions of our work can be summarized as follows : • We propose a novel and efficient text instance representation method named CentripetalText , in which text instances are decomposed into the combination of text kernels and centripetal shifts . The attached post-processing algorithm is also simple and robust , making the generation of text contours fast and accurate . • To reduce the burden of model training , we develop an enhanced loss function , namely the Relaxed L1 Loss , mainly for dense centripetal shift regression , which further improves the detection performance . • Equipped with the proposed CT and the Relaxed L1 Loss , our scene text detector achieves superior or competitive results compared to other existing approaches on the curved or oriented text benchmarks , and our end-to-end scene text recognizer surpasses the current state of the art . 2 Related work . Text instance representation methods can be roughly classified into two categories : component-free methods and component-based methods . Component-free methods treat every text instance as a complete whole and directly regress the rotated rectangles or quadrangles for describing scene texts without any reconstruction process . These methods are usually inspired by general object detectors such as Faster R-CNN [ 29 ] and SSD [ 19 ] , and often utilize heuristic anchors as prior knowledge . TextBoxes [ 15 ] successfully adapted the object detection framework SSD for text detection by modifying the aspect ratios of anchors and the kernel scales of filters . TextBoxes++ [ 14 ] and EAST [ 51 ] could predict either rotated rectangles or quadrangles for text regions with and without the prior knowledge of anchors , respectively . SPCnet [ 40 ] modified Mask R-CNN [ 7 ] by adding the semantic segmentation guidance to suppress false positives . Component-based methods prefer to model text instances from local perspectives and decompose instances into components such as characters or text fragments . SegLink [ 31 ] decomposed long texts into locally-detectable segments and links , and combined the segments into whole words according to the links to get final detection results . MSR [ 43 ] detected scene texts by predicting dense text boundary points . BR [ 30 ] further improved MSR by regressing the positions of boundary points from two opposite directions . PSENet [ 37 ] gradually expanded the detected areas from small kernels to complete instances via a progressive scale expansion algorithm . 3 Methodology . In this section , we first introduce our new representation ( i.e. , CT ) for texts of arbitrary shapes . Then , we elaborate on our method and training details . 3.1 Representation . Overview An efficient scene text detector must have a well-defined representation for text instances . The traditional description methods inherited from generic object detection ( e.g. , rotated rectangles or quadrangles ) fail to encode the geometric properties of irregular texts . To guarantee the flexibility and generality , we propose a new method named CentripetalText , in which text instances are composed of text kernels and centripetal shifts . As demonstrated in Fig . 2 , our CT expresses a text instance as a cluster of the pixels which can be shifted into the region of the same text kernel through centripetal shifts . As pixels are the basic units of digital images , CT has the ability to model different forms of text instances , regardless of their shapes and lengths . Mathematically , given an input image I , the ground-truth annotations are denoted as { T1 , T2 , .. , Ti , .. } , where Ti stands for the ith text instances . Each text instance Ti has its corresponding text kernel Ki , a shrunk version of the original text region . Since a text kernel is a subset of its text instance , which satisfies Ki ✓ Ti , we treat it as the main basic of the pixel aggregation . Different against the distance in conventional methods , each centripetal shift sj which appears in a position pj of the image guides the clustering of text pixels . In this sense , the text instance Ti can be easily represented with the aggregated pixels which can be shifted into the region of a text kernel according to the values of their centripetal shifts : Ti = { pj | ( pj + sj ) 2 Ki } . ( 1 ) Label generation The label generation for the probability map is inspired by PSENet [ 37 ] , where the positive area of the text kernel ( shaded areas in Fig . 3 ( b ) ) is generated by shrinking the annotated polygon ( shaded areas in Fig . 3 ( a ) ) using the Vatti clipping algorithm [ 35 ] . The offset of shrinking is computed based on the perimeter and area of the original polygon and the shrinking ratio is set to 0.7 empirically . Since the annotations in the dataset may not perfectly fit the text instances well , we design a training mask M to distinguish the supervision of the valid and ignoring regions . The text instance excluding the text kernel ( Ti Ki ) is the ignoring region , which means that the gradients in this area are not propagated back to the network . The training mask M can be formulated as follows : Mj = ( 0 , if pj 2 S i ( Ti Ki ) 1 , otherwise . ( 2 ) We simply multiply the training mask by the loss of the segmentation branch to eliminate the influence brought by wrong annotations . In the label generation step of the regression branch , the text instance affects the centripetal shift map in three ways . First , the centripetal shift in the background region ( I S i Ti ) should prevent the background pixels from entering into any text kernel and thus we set it to ( 0 , 0 ) intuitively . Second , the centripetal shift in the text kernel region ( S i Ki ) should keep the locations of text kernel pixels unchanged and we also set it to ( 0 , 0 ) for convenience . Third , we expect that each pixel in the region of the text instance excluding the text kernel ( S i ( Ti Ki ) ) can be guided to its corresponding kernel by the centripetal shift . Therefore , we continuously conduct the erosion operation over the text kernel map twice , compare these two temporary results and obtain the text kernel reference ( polygons with solid lines in Fig . 3 ( c ) ) as the destination of the generated centripetal shift . As shown in Fig . 3 ( d ) , we build the centripetal shift between each pixel in the shaded area and its nearest text kernel reference to prevent numerical accuracy issues caused by rounding off . Note that if two instances overlap , the smaller instance has the higher priority . Specifically , the centripetal shift sj can be calculated by : sj = 8 > > > < > > > : ( 0 , 0 ) , if pj 2 ( I S i Ti ) ! pjp⇤j , if pj 2 S i ( Ti Ki ) ( 0 , 0 ) , if pj 2 S i Ki , ( 3 ) where p⇤j represents the nearest text kernel reference to the pixel pj . During training , the Smooth L1 loss [ 5 ] is applied for supervision . Nevertheless , according to a previous observation [ 11 ] , the dense regression can be modeled as a simple Dirac delta distribution , which fails to consider the ambiguity and uncertainty in the training data . To address the problem , we develop a regression mask R for the relaxation operation and integrate it into the Smooth L1 loss to reduce the burden of model training . We extend the correct prediction from one specific value to a range and any centripetal shift which moves the pixel into the right region is treated as positive during training . The regression mask can be formulated as follows : Rj = 8 > < > : 0 , if pj 2 Ti and ( pj + bsj ) 2 Ki or pj 62 S i Ti and ( pj + bsj ) 62 S i Ki 1 , otherwise . ( 4 ) where bsj and sj denote the predicted centripetal shift at the position j and its ground truth , respectively . Like the segmentation loss , we multiply the regression mask by the Smooth L1 loss and form a novel loss function , namely the Relaxed L1 loss for dense centripetal shift prediction , to further improve the detection accuracy . The Relaxed L1 loss function can be formulated as follows : Lregression = X j Rj · SmoothL1 ( sj , bsj ) , ( 5 ) where SmoothL1 ( ) denotes the standard Smooth L1 loss .

Summary: Update after discussion among reviewers: I am still strongly in favor of accepting this paper.   The paper presents a new method to represent arbitrary shaped text in images and shows how this representation can be used in the output layer of some of the recent text detection methods.   The results look qualitatively and quantitative great.   The method uses a simple representation that is intuitively understandable.