PolarNet: Learning to Optimize Polar Keypoints for Keypoint Based Object Detection

1 INTRODUCTION . Deep learning based object detection techniques have achieved remarkable success in many real-world applications ( Krizhevsky et al. , 2012 ; He et al. , 2016 ; Goodfellow et al. , 2016 ) . The mainstream stateof-the-art detectors are often based on the anchor-based detection methods ( Ren et al. , 2015 ; Girshick , 2015 ; Lin et al. , 2017b ) , which heavily rely on the design and selection of appropriate anchor boxes , namely a set of predefined bounding boxes of a certain height and width , to capture various scales and aspect ratios of different object classes for detection . Unlike the anchor-based detectors , the anchor-free detectors have emerged recently as a promising direction for object detection that eliminates the need of manually designing anchor boxes ( Zhu et al. , 2019 ; Tian et al. , 2019 ; Law & Deng , 2018 ; Duan et al. , 2019 ) . In literature , a variety of anchor-free object detectors have been proposed based on different object modeling strategies . Figure 1 ( a ) - ( e ) gives examples comparing five popular anchor-free detectors from the perspective of object modeling . For example , CornerNet ( Law & Deng , 2018 ) was proposed for detecting objects using a pair of corner points . Instead of using two corners , CenterNet ( Zhou et al. , 2019a ) proposed modeling an object as one center point of its bounding box . Besides these , there are also a number of other anchor-free detectors that extend these ideas of Corner-based or Centerness-based or various other keypoint design strategies to improve the detection performance . FSAF ( Zhu et al. , 2019 ) and FCOS ( Tian et al. , 2019 ) predict objects by learning the offsets to the boundary from sampled keypoints . FCOS ( Tian et al. , 2019 ) uses many keypoints by treating every pixel as a keypoint , while FSAF ( Zhu et al. , 2019 ) samples a set of multiple keypoints from the center region to eliminate points near the boundary . Among keypoint based object detection , two different strategies are commonly adopted . One is keypoint position based ( determining the bounding box by the position of the keypoints ) such CornerNet ( Law & Deng , 2018 ) and CenterNet ( Duan et al. , 2019 ) . The other is keypoint offsets based ( determining the bounding box by learning offsets from keypoints ) such as FSAF ( Zhu et al. , 2019 ) , CenterNet ( Zhou et al. , 2019a ) and FCOS ( Tian et al. , 2019 ) . Compared to the keypoint position based methods , keypoint offset based methods are able to better capture contextual information about the objects , and are thus more commonly used to design detectors . However , existing object modeling strategies for keypoint offset based methods may be sub-optimal . Most existing anchor-free detectors such as FCOS ( Tian et al. , 2019 ) are based on Cartesian coordinates , which learn offsets to the boundary of objects . However , this kind of design yields a lot of poor quality keypoints . These points are near the boundary with extremely large variance of offsets ( See Figure 1 ( e ) ) . Besides , the prediction heads are also based on the scale-sensitive features , which further increases the optimization difficulties . Our goal is to have an anchor-free detector , which can avoid poor quality keypoints , and is able to simultaneously learn scale-insensitive feature for object prediction . To achieve this goal , in this paper , we propose a new keypoint based object detector named “ PolarNet ” , which learns keypoints based on polar coordinates . Figure 1 ( f ) illustrates the idea of the proposed PolarNet compared to the other anchor-free detectors . The set of keypoints is represented by polar coordinates to avoid large variance of offsets . And the features of corner points in PolarNet are also used to localize objects , which is scale-invariant . The key contributions of this work are : • We introduce a unified view of keypoint based object detection for understanding popular anchor-free object detectors , in which many popular anchor-free object detectors can be viewed as a special case of keypoint based detectors with different object modeling strategies ; • We propose a new anchor-free object detector named “ PolarNet ” which presents keypoints based on polar coordinates which enables learning of better quality keypoints by reducing the variance of learned offsets compared to the existing approaches . • We conduct experiments to evaluate the performance of our PolarNet detector on the COCO benchmark , in which our results show that PolarNet outperforms all the existing anchor-free detectors , and is able to achieve highly competitive results better or on par with the stateof-the-art two-stage anchor-based detectors on COCO test-dev ( 47.8 % and 50.3 % AP with DCNv2-ResNeXt-101 backbone on COCO test-dev under single-model single-scale and multi-scale settings ) . The rest of this paper is organized as follows . Section 2 reviews two major categories of related work in deep-learning based object detection : popular anchor-based detectors and recent anchor-free detectors . Section 3 presents the proposed PolarNet detector in detail . Section 4 discusses our experimental results and analysis , and Section 5 concludes this paper . 2 RELATED WORK . In this section , we briefly review two major groups of related work in the literature of deep-learning based object detection approaches : the mainstream family of anchor-based object detection methods and the emerging family of anchor-free object detection methods . We refer readers to more extensive surveys of deep-learning based object detection studies in ( Liu et al. , 2020 ; Wu et al. , 2020 ) . 2.1 ANCHOR-BASED OBJECT DETECTION . The methods in this group represent the mainstream detectors widely used in many real-world applications . They can be broadly categorized into two groups : two-stage detectors ( Ren et al. , 2015 ; Girshick et al. , 2014 ; Girshick , 2015 ) and one-stage detectors ( Liu et al. , 2016 ; Zhang et al. , 2018 ; Lin et al. , 2017b ) . Two-stage detectors consist of two stages : ( i ) region proposal generation , and ( ii ) region proposal classification and regression . For example , one of most popular two-stage detectors is Faster R-CNN ( Ren et al. , 2015 ) that uses a Region Proposal Network ( RPN ) to generate regions of interest in the first stage and then sends the region proposals down the pipeline for object classification and bounding-box regression . Faster R-CNN has resulted in many various extensions and improvements in literature ( Lin et al. , 2017a ; Dai et al. , 2017 ; 2016 ) . Single-stage detectors , also known as single-shot detectors , do not need the proposal generation and simply treat object detection as a simple classification and regression problem by taking an input image and directly learning the class probabilities and bounding box coordinates of objects via convolutional networks . Popular single-stage detectors include YOLO ( Redmon et al. , 2016 ) , SSD ( Liu et al. , 2016 ) , and RetinaNet ( Lin et al. , 2017b ) . Typically , single-stage detectors are less accurate than two-stage detectors , but they are much faster and thus more amenable to real-time inference needs . In general , anchor-based detectors suffer from some critical limitations , including requiring heuristic design of anchors , poor alignment of anchors with ground truth objects , and incurring a large number of false positives when anchors are not carefully designed . 2.2 ANCHOR-FREE OBJECT DETECTION . Here , we view most anchor-free detectors from as keypoint based detectors . We can categorize the existing anchor-free object detection methods into two groups according to different bounding box prediction strategies : 1 ) Keypoint Position and 2 ) Keypoint Offsets based detectors . We review some representative works in each group below . Keypoint Position : This kind of anchor-free detectors predict bounding box via keypoint positions , such as corners . A representative detector is CornerNet ( Law & Deng , 2018 ) . CornerNet models each object by a pair of corner keypoints , which eliminates the need of anchor boxes and is perhaps the first anchor-free detector that achieved the state-of-the-art single-stage object detection accuracy . There have been some extensions of CornerNet to improve its efficiency towards real-time applications such as CornerNet-Lite ( Law et al. , 2019 ) . Based on CornerNet , CenterNet ( Duan et al. , 2019 ) , models an object by a triplet of keypoints , including one center and two corner keypoints . ExtremeNet ( Zhou et al. , 2019b ) models an object by a set of five keypoints , including one center and four extreme points ( top-most , leftmost , bottom-most , right-most ) of an object based on a standard keypoint estimation network . RPDet ( Yang et al. , 2019 ) represents an object by a fixed set of 9 keypoints learned from the center of an object which are refined progressively during the training process . RPNet V2 ( Chen et al. , 2020 ) fuses the corner and foreground features in into the original RPNet to detect objects . Their work is in parallel with ours and shares some similarities with our corner learning module . However , our PolarNet is mainly focused on the design of the polar representation to handle scale issues based on vanilla FCOS , and the corner learning module is an auxiliary module to further enhance the feature representation of the detector . Keypoint Offsets : This kind of anchor-free detectors predict bounding box via learning offsets based on keypoints . A representative anchor-free detector is CenterNet ( Zhou et al. , 2019a ) that models an object by the center point of its bounding box , and it uses keypoint estimation to find center points and regresses to all other object properties , such as size , 3D location , orientation , and even pose . Some approaches such as FSAS ( Zhu et al. , 2019 ) and FoveaBox ( Kong et al. , 2020 ) sample many keypoints from the center region of an object and learn to predict objects . Some approaches such as FCOS ( Tian et al. , 2019 ) and SAPD ( Zhu et al. , 2020 ) treat all the pixels within an object as candidate keypoints during training and improve them by a separate post-processing strategy . These keypoint based detectors all learn offsets to the boundary of the objects based on Cartesian coordinates , which yields a lot of poor quality keypoints . Some strategies are adopted to remove these bad points such as center sampling ( Kong et al. , 2020 ; Zhu et al. , 2019 ) or centerness score ( Tian et al. , 2019 ) . Unlike the above keypoint based detectors that learn keypoints based on Cartesian coordinates , our approach learns a set of keypoints based on polar coordinates which makes it easier to learn high quality keypoints . 3 POLARNET DETECTOR . 3.1 POLAR COORDINATES VS CARTESIAN COORDINATES Many popular keypoint based detectors represent keypoints based on Cartesian coordinates . This strategy yields a lot of keypoints with poor quality ( see Figure 1 ( e ) ) , making it difficult to optimize without manually designed strategies ( such as centerness ( Tian et al. , 2019 ) or center sampling ( Zhu et al. , 2019 ) ) . A better approach for keypoints representation is thus required to avoid poor quality keypoints ( such as points near the boundary ) . In order to overcome this challenge , we represent the keypoints with polar coordinates instead of vanilla Cartesian coordinates . We learn the offsets of each keypoints pointing to the corners , which significantly reduce the scale variance of keypoints . Furthermore , the features of corners are also used for scale-insensitive localization . For each keypoint , we predict offsets based on this point to generate the bounding box . Vanilla keypoint based detectors learn keypoints based on Cartesian coordinates , which learns 4D vectors ( t , l , b , r ) , corresponding to ( top , bottom , left , right ) pointing to the boundary of the objects ( See Figure 2 ( a ) ) . In PolarNet , for each object , we learn the offsets of keypoints which point to the corners ( In this paper we use top left ( tl ) and bottom right ( br ) as the target corner pair ) . And thus for each point ( x , y ) , we predict a pair of 4D vectors : ( θtl , ρtl , θbr , ρbr ) , which point to the corners of the object ( See Figure 2 ( b ) ) . Here ρ and θ denote the distance and sine value of angle of point ( x , y ) to corner . We defer the discussion for training the keypoint predictor later . For keypoint offsets based detectors , learning offsets with large variance yields unsatisfactory results . And thus in many existing methods , poor quality points will be removed ( Zhu et al. , 2019 ) or be suppressed ( Tian et al. , 2019 ) during training . Here we analyze the scale variance of the learned offsets of the keypoints in FCOS and in PolarNet during training . Assume in FCOS and PolarNet , the learned offsets of keypoint ( x , y ) are represented as Bx , y . To show the scale variance of learned offsets , We plot the heatmaps of scale variance of the learned offsets of FCOS and PolarNet in Figure 2 ( c ) and ( d ) respectively as : k = min { Bx , y } max { Bx , y } ( 1 ) where k is the value plotted in heatmaps . In Figure 2 ( c ) , nearly all the keypoints within the objects present a large scale variance except the ones within the central region . Specifically when the point is close to the boundary , the variance of learned offsets is very large . In Figure 2 ( d ) , the majority of the keypoints in PolarNet learn offsets with much smaller scale variance compared with Figure 2 ( c ) , which significantly reduce the optimization difficulties .

Summary: This paper proposes an anchor-free object detector that does bounding box regression in the polar coordinate instead of in the Cartesian coordinate. The motivation of doing this is because there are larger variance in offset vectors in the Cartesian coordinate (the extreme case when a point is on one of the four corners of the bounding box, the offset vector becomes [0, w, 0, h]). The authors propose a solution to regress to the pair of corners (either TL+BR or TR+BL) in the polar coordinate, and select the corner pair that gives the smallest variance during training.