PolarNet: Learning to Optimize Polar Keypoints for Keypoint Based Object Detection
Authors: Wu Xiongwei
Doyen Sahoo
Steven HOI
Venue: ICLR 2021 Poster (2021)
Keywords: Object Detection
Deep Learning

Abstract: A variety of anchor-free object detectors have been actively proposed as possible alternatives to the mainstream anchor-based detectors that often rely on complicated design of anchor boxes. Despite achieving promising performance on par with anchor-based detectors, the existing anchor-free detectors such as FCOS or CenterNet predict objects based on standard Cartesian coordinates, which often yield poor quality keypoints. Further, the feature representation is also scale-sensitive. In this paper, we propose a new anchor-free keypoint based detector ``PolarNet", where keypoints are represented as a set of Polar coordinates instead of Cartesian coordinates. The ``PolarNet" detector learns offsets pointing to the corners of objects in order to learn high quality keypoints. Additionally, PolarNet uses features of corner points to localize objects, making the localization scale-insensitive. Finally in our experiments, we show that PolarNet, an anchor-free detector, outperforms the existing anchor-free detectors, and it is able to achieve highly competitive result on COCO test-dev benchmark ($47.8\%$ and $50.3\%$ AP under the single-model single-scale and multi-scale testing) which is on par with the state-of-the-art two-stage anchor-based object detectors. The code and the models are available at https://github.com/XiongweiWu/PolarNetV1

Published as a conference paper at ICLR 2021
POLARNET: LEARNING TO OPTIMIZE POLAR KEY- POINTS FOR KEYPOINT BASED OBJECT DETECTION
Xiongwei Wu', Doyen Sahoo?, Steven C.H. Hoi!.
"Singapore Management University
Salesforce Research Asia
xwwu3 edu. 3g
{dsahoo, shoi choil@ssaeesforee.co
ABSTRACT
A variety of mnchor-free object detectors have been actively proposed as possible al- ternatives the mainstream anchor-based detectors that often rely on complicated design of anchor boxes. Despite achieving promising performance on par with anchor-based detectors, the existing anchor-free detectors such FCOS or Center- Net predict objects based on standard Cartesian coordinates, which often yield poor quality keypoints. Further, the feature representation is also scale-sensitive In this paper, we propose new anchor-free keypoint based detector "PolarNet", where keypoints are represented as set of Polar coordinates instead of Cartesian coor- dinates. The "PolarNet" detector learns offsets pointing to the corners of objects in order to learn high quality keypoints. Additionally, PolarNet uses features corner points to localize objects, making the localization scale-insensitive. Finally in our experiments, we show that PolarNet, an anchor-fres detector, outperforms the existing anchor-free detectors, and it is able achieve highly competitive result on COCO test-dev benchmark (47.8% and 50.3% AP under the single-model single-scale and multi-scale testing) which is on par with the state-of-the-ar two- stage anchor-basee object detectors. The code and the models are available at https: /tt:://gitttne wwt.btttt.uu...go/gg com Xi http:/xxooggwwiiicc wwwaauuaooelee
INTRODUCTION
Deep learning based object detection techniques have achieved remarkable success in many real-world applications (Krizhevsky et al. 2012 He et al. 20166] Goodfellow et al. 2016). The mainstream state- of-the-art detectors are often based on the anchor-based detection methods (Ren et al. 2015 Girshick 2015; Lin et al. 2017b), which heavily rely on the design and selection of appropriate anchor boxes, namely set of predefined bounding boxes of certain height and width, capture various scales and aspect ratios of different object classes for detection. Unlike the anchor-based detectors, the anchor-free detectors have emerged recently promising direction for object detection that
eliminates the need of manually designing anchor boxes (Zhu et al. 2019 Tian et al. 2019 Law &
Deng 2018 Duan et 2019).
In literature, a variety of anchor-free object detectors have been proposed based on different object modeling strategies. Figure (a)-(e) gives examples comparing five popular anchor-free detectors from the perspective of object modeling. For example, CornerNet (Law & Deng 2018) was proposed for detecting objects using a pair of corner points. Instead of using two corners, CenterNet (Zhou et al. 2019a) proposed modeling an object as one center point of its bounding box. Besides these, there are also a number of other anchor-fre free detectors that extend these ideas of Corner-based or Centerness- based various other keypoint design strategies to improve the detection performance. FSAF (Zhu et al.) 2019) and FCOS (Tian et al. 2019) predict objects by learning the offsets to the boundary from sampled keypoints. FCOS ( Tian et al. 2019) uses many keypoints by treating every pixel as a keypoint, while FSAF (Zhu et al.) 2019) samples a set of multiple keypoints from the center region to eliminate points near the boundary.
Among keypoint based object detection, two different strategies are commonly adopted. One is keypoint position based (determining the bounding box by the position of the keypoints) such

---

Published as a conference paper at ICLR 2021
(a) CornerNet Law & Deng (b) CenterNet Zhou et al. 2019a
(c) RPDet Yang et al.
2019)
2018
(d) FSAF Zhu et al. 2019)
(e) FCOS (Tian et al. 2019
(I) PolarNet (ours)
Figure 1: Comparison of different anchor-free object detection methods. Red dots denote positive keypoints and grey dots denote negative keypoints.
2019p. p. The other is keypoint offsets based (determining the bounding box by learning offsets from keypoints) such as FSAF (Zhu et al. (2019). CenterNet (Zhou et al. 2019a and FCOS (Tian et al., 2019). Compared to the keypoint position based methods, keypoint offset based methods are able better capture contextual information about the objects, and are thus more commonly used to design detectors.
CornerNet (Law & Deng 2018 and CenterNet (Duan et al.
However, existing object modeling strategies for keypoint offset based methods may be sub-optimal. Most existing anchor-free detectors such as FCOS (Tian et al. 2019) are based on Cartesian coordi- nates, which learn offsets to the boundary of objects. However, this kind of design yields lot of poor quality keypoints. These points are near the boundary with extremely large variance of offsets (See Figure Te)). Besides, the prediction heads are also based on the ceale-sensitive features, which further increases the optimization difficulties. Our goal is to have an anchor-fres detector, which can avoid poor quality keypoints, and is able to simultaneously learn scale-insensitive feature for object prediction. To achieve this goal, in this paper, we propose new keypoint based object detector named "PolarNet", which learns keypoints based on polar coordinates. Figure (f) illustrates the idea of the proposed PolarNet compared to the other anchor-fiee detectors. The set of keypoints is represented by polar coordinates to avoid large variance of offsets. And the features corner points in PolarNet are also used to localize objects, which is scale-invariant.
The key contributions of this work are:
We introduce a unified view of keypoint based object detection for understanding popular anchor-free object detectors, in which many popular anchor-free object detectors can be viewed as special case of keypoint based detectors with different object modeling strategies;
anchor-free object detector named "PolarNet" which presents keypoints based on polar coordinates which enables learning better quality keypoints by reducing the variance of learned offsets compared to the existing approaches.
We propose
We conduct experiments to evaluate the performance of our PolanNet detector on the COCO benchmark, in which our results show that PolarNet outperforms all the existing anchor-free detectors, and is able to achieve highly competitive results better or on par with the state- of-the-art two-stage anchor-based detectors on COCO test-dev (47.8% and 50.3% AP with DCNv2-ResNeXX backbone on COCO test-dev under single-moder single-scale and multi-scale settings).
2

---

Published as a conference paper at ICLR 2021
The rest of this paper is organized as follows. Section reviews two major categories of related work in deep-learning based object detection: popular anchor-basee detectors and recent anchor- mnchor-free detectors. Section 3 presents the proposed PolarNet detector in detail. Section discusses our experimental results and analysis, and Section concludes this paper.
RELATED WORK
In this section, we briefly review two major groups of related work in the literature of deep-learning based object detection approaches: the mainstream family of anchor-based object detection methods and the emerging family anchor-free object detection methods. We refer readers to more extensive surveys of deep-learning based object detection studies in (Liu et al. 2020: Wu et al. 2020
2.1
ANCHOR-BAR -BASE OBJECT DETECTION
The methods in this group represent the mainstream detectors widely used in many real-world applications. They can be broadly categorized into two groups: two-stage detectors (Ren al. 2015 Girshick et al., 2014 Girshick 2015) and one-stage detectors (Liu et al.) 2016 Zhang et al.
2018 Lin et al. (2017)). 76). Two-stage detectors consist of two stages: (i) region proposal generation, and (ii) region proposal classification and regression. For example, one of most popular two-stage detectors is Faster R-CNN (Ren et al.) 2015) that uses a Region Proposal Network (RPN) generate regions of interest in the first stage and then sends the region proposals down the pipeline for object classification and bounding-box regression. Faster R- R-CNN has resulted many various extensions and improvements in literature (Lin et al., 2017a Dai et al. 2017 (016). Single-stage detectors, also known as single-shot detectors, do not need the proposal generation and simply treat object detection as simple classification and regression problem by taking input image and directly learning the class probabilities and bounding box coordinates of objects via convolutional networks. Popular single-stage detectors include YOLO (Redmon et al., 2016), SSD (Liu et al. (016), and RetinaNet (Lin et al. 201 Typically, single-stage detectors are less accurate than two-stage detectors, but they much faster and thus more amenable to real-time inference needs. In general, anshor-based detectors suffer from some critical limitations, including requiring heuristic design of anchors, poor alignment of anchors with ground truth objects, and incurring large number of false positives when anchors are not carefully designed.
2.2 ANCHOR-FREE OBJECT DETECTION
Here, we view most anchor-free detectors from as keypoint based detectors. We can categorize the existing anchor-free object detection methods into two groups according to different bounding box prediction strategies: 1) Keypoint Position and 2) Keypoint Offsets based detectors. We review some representative works in each group below.
Keypoint Position:
This kind of anchor-fin frreer--reeeae detectors predict bounding box via keypoint positions, such as corners. A representative detector is CornerNet (Law & Deng 2018). CornerNet models cach object by pair of corner keypoints, which eliminates the need of anchor boxes and is perhaps the first anchor-free free detector that achieved the state-of-the-an single-stage object detection accuracy. There have been some extensions of CornerNet to improve its efficiency towards real-time applications such as CornerNet-Lite (Law et al. 2019). Based on CornerNet, CenterNet (Duan et al. 2011). models an object by a triplet of keypoints, including one center and two corner keypoints. ExtremeNet (Zhou et al. 2019b) models an object by set of five keypoints, including one center and four extreme points (top-most, leftmost, bottom-man
right-most) of an object based on standard keypoint estimation network. RPDet (Yang et al. 2019) represents an object by fixed set of 9 keypoints learned from the center of an object which are refined progressively during the training process. RPNet V2 (Chen et al. 2020) fuses the corner and foreground features in into the original RPNet to detect objects. Their work is in parallel with ours and shares some similarities with our corner learning module. However, PolarNet is mainly focused the design of the polar representation to handle scale issues based on vanilla FCOS, and the corner learning module is an auxiliary module to further enhance the feature representation of the detector.
This kind of anchor-fit free detectors predict bounding box via learning offsets based on keypoints. A representative anchor-free detector is CenterNet (Zhou al. 2019a) that models an
Keypoint Offsees:
3

---

Published as a conference paper at ICLR 2021
object by the center point of its bounding box, and it uses keypoint estimation to find center points and regresses to all other object properties, such as size. 3D location, orientation, and even pose. Some approaches such as FSAS (huu et al., 2019) and FoveaBox (Kong et al., 2020) sample many keypoints from the center region of an object and learn to predict objects. Some approaches such as FCOS (Tian et al. 2019) and SAPD (Zhu al.) 2020) treat all the pixels within an object as candidate keypoints during training and improve them by a separate post-processing strategy. These keypoint based detectors all learn offsets to the boundary of the objects based on Cartesian coordinates, which yields lot of poor quality keypoints, Some strategies are adopted to remove these bad points such as center sampling (Kong et al., 2020 Zhu et al. 2019) or centerness score (Tian al., (019). Unlike the above keypoint based detectors that learn keypoints based on Cartesian coordinates, our approach learns a set of keypoints based on polar coordinates which makes easier to learn high quality keypoints.
POLARNET DETECTOR
3.1
POLAR COORDINATES VS CARTESIAN COORDINATES
(a) Cartesian
(b) Polar
(c) Heatmap of FCOS (d) Heatmap of PolarNet
Figure 2: (a) and (b) show the comparison of Cartesian coordinates and Polar coordinates. (c) and (d) show the heatmap of learned offsets in training FCOS and PolarNet. In (c), the majority of keypoints learned by FCOS have large scale variance, indicating the necessity to suppress the poor quality keypoints during training. In (d), the scale variance of keypoints in PolarNet is much smaller than FCOS, and thus more high quality keypoints can learned during training.
Many popular keypoint based detectors represent keypoints based on Cartesian coordinates. This strategy yields a lot of keypoints with poor quality (see Figure Te)). making it difficult to optimize without manually designed strategies (such as centerness (Tian al. 2019) or center sampling (Zhu et al. 2019)). A better approach for keypoints representation is thus required to avoid poor quality keypoints (such as points near the boundary). In order to overcome this challenge, we represent the keypoints with polar coordinates instead of vanilla Cartesian coordinates. We learn the offsets of each keypoints pointing to the corners, which significantly reduce the scale variance of keypoints. Furthermore, the features of corners also used for ialeeinsenne localization.
For each keypoint, we predict offsets based on this point to generate the bounding box. Vanilla keypoint based detectors learn keypoints based Cartesian coordinates, which learns 4D vectors b, r). corresponding to (top, bottom, left, right) pointing to the boundary of the objects (See Figure 2 a)). In PolarNet, for each object, we learn the offsets of keypoints which point the corners (In this paper we use top left (tl) and bottom right (br) as the target corner pair). And thus for each point (z, 3). we predict pair of 4D vectors: (0a, Pa, Pbe, Por). which point to the corners of the object (See Figure Here and 0 denote the distance and sine value of angle of point (I, y) to corner. We defer the discussion for training the keypoint predictor later.
For keypoint offsets based detectors, learning offsets with large variance yields unsatisfactory results. And thus in many existing methods, poor quality points will be removed (Zhu et al.) 2019) or be suppressed (Tian et al.) 2019) during training. Here we analyze the scale variance of the learned offsets of the keypoints in FCOS and in PolarNet during training. Assume in FCOS and PolarNet, the learned offsets of keypoint (I, y) are represented as Bx,y. To show the scale variance of learned offsets, We plot the heatmaps of scale variance of the learned offsets of FCOS and PolarNet in Figure 2(c) and (d) respectively as:
k = min Bx.y> } max Bx.y)
(I)
4

---

Published as a conference paper at ICLR 2021
Corner Leaniing
p.0
CNN
Image
Feature map
AutoSample
Figure 3: The architecture of PolarNet detector. Following the CNN backbone, also apply the Feature Pyramid Network (FPN). The objectness prediction module is pixel-wise prediction to predict the likelihood of relevant object in the location of the pixel. Each keypoint is represented by polar coordinates which points to corners. The optimal coordinates are selected for training and the features of corners are also used for training detectors.
where k is the value plotted in heatmaps. In Figure 2c), nearly all the keypoints within the objects present a large scale variance except the ones within the central region. Specifically when the point close to the boundary, the variance of learned offsets is very large. In Figure d)), the majority of the keypoints PolarNet learn offsets with much smaller seale variance compared with Figure which significantly reduce the optimization difficulties.
3.2 OVERVIEW
We now present the proposed keypoint based detection network (PolarNet), a new anchor-free detector that is capable of better representing keypoints and localizing the objects. Figure gives an overview of the architecture of the proposed PolarNet. An input image is passed through a CNN network to produce a set of feature maps. Note that following the CNN backbone, we also apply the Feature Pyramid Network (FPN) to help deal with scale variances. Based on the feature maps, Fully Convolutional layer is applied to perform the pixel-wise objectness prediction, which predicts the objectness likelihood of a pixel with respect within the objects, we will learn a set of polar coordinates pointing to the corners of the objects. The keypoints selector will be used here to select the optimal coordinates for detector training. And the features of corners will also be used here to supervise the detector learning. Finally, after the selected keypoints are obtained, the keypoints' bounding boxes together with their objectness and likelihood scores are passed through a Non-Maximum Suppression (NMS) to obtain the final detection result.
particular object category. Then for each keypoint
Next we will discuss several key modules of the proposed framework, including polar coordinates, pixel-wise objectness prediction, overall detector training, inferences, and other details.
3.3 PIXEL-WISE OBJECTNESS PREDICTION
The input to this module is a set of feature maps from a CNN backbone, denoted by F; HXXXC the feature map layer of CNN with a total of C classes. We view any pixel/location (æ, of the feature map a potential candidate for keypoints, and the objectness prediction module aims to predict how likely a particular location of the feature map is relevant to some particular class of objects. This idea follows the principle of fully convolutional networks (FCN) for semantic segmentation and has also been used previously in FCOS (Tian et al. 2019).
Specifically, we predict the objectness of particular location (ær, v) by real vector (Car,ys br.y), where Ca.y C) denotes C-dimensional vector of object class prediction scores for (z, y).
3.4 DETECTOR TRAINING
3.4.1 TRAINING LOSS FOR KEYPOINTS CLASSIFICATION
We train the objectness prediction using both classification loss and bounding box regression loss. We define Q (x, y) is the of candidate locations that fall into the ground-truth bounding boxes.
5

---

Published as a conference paper at ICLR 2021
The classification loss is defined as
Lelss
I Trocal (or,ys Cx.y)
(x.9)
where Liecal based on the focal loss Lin et al. 2017b) and ci E 1Jc denotes the ground-truth class labels at location (x, y). For bounding box regression, we only consider a location that falls into any ground-truth box. Unlike vanilla keypoint detectors which yield poor quality keypoints due to large variance of learned offsets, in our model the majority of keypoints have similar distance p to the keypoints and thus it is more suitable for training the detectors. We still further improve the detector algorithms with the keypoints sampling methods, which removes the remaining poor quality keypoints around the corners. Here validate few sampling strategies and select simple yet effective strategy, AutoSample in our final model. Please refer Appendix .1 for more analysis.
3.4.2 TRAINING LOSS FOR CORNER FEATURE LEARNING
In vanilla keypoint detector, the learned features for prediction are mainly based on FPN and are sensitive to scales. Inspired by the scale-invariant property of corner features, we aim to enhance the original feature representations based the corner features. For each feature map of the feature pyramid, we first learn the corner feature supervised by the corner annotation, and then integrate the learned corner feature into the original feature map to make it robust to scale variance. We follow the similar training strategy and define the corner loss Loumer as CornerNet (Law Deng (018) and we learn detectors based on the new generated feature maps. Please refer Appendix for more details.
3.4.3 TRAINING LOSS FOR LOCALIZATION
Here we train the bbox regressors based on the new generated features maps. Suppose the polar coordinates in (r. y) is Bx.y as (Obe: Par: Qg. Pt), We can easily obtain the bounding box based on the polar coordinate. And we define the bounding box regression loss as:
Labox
Σ (x.N)EQ
Laiol (Br,y. B
(3)
Q
where LEBU based the GloU loss (Rezatofighi et al. 2019). We transform the learned offsets into bounding box and optimize the GloU loss. In addition to GloU loss, here we also learn the supervision from corners
Lhheta
Σ (2.9) (d(()
LeE(sin Oy, sin sin 07)
(4)
2
where LCE denotes cross entropy loss. We find obtaining the supervision from 0 is sufficient to train the model and thus we do not learn supervision from p. Finally, we define localization loss as:
Lisc
Labux + Lihetu
3.4.4 OVERALL TRAINING LOSS AND ADAPTIVE KEYPOINT TRAINING
Combining the above training losses for both objectness prediction and polar coordinates modules, we can define the overall training loss function as follows:
Loverall = Leets + Lise + Leonner
(6)
3.5 INFERENCE
During the inference stage. PolarNet takes an image as input and passes it through the CNN network followed by the FPN module to obtain the feature maps. Based on the resulting feature maps, we then predict the objectness scores Cz.y and bounding box offsets b of each point/location (x, v) on the feature map. After that, we select top-k points/locations with the highest class objectness scores the candidate points to the keypoint prediction module. Our model shares the same computation cost as vanilla FCOS except the corner learning module. Finally, we pass the predicted boxes to a Non- Non-Maximum Suppression (NMS) to obtain the final detection result.

---

Published as a conference paper at ICLR 2021
4 EXPERIMENTS
4.1 EXPERIMENTAL DATASET AND SETUP
We conducted experiments on MSCOCO 2017 dataset, which has 80 categories three splits: train (115k images), val (5k images), and test-dev (20k images). Following common practice, used the train set to train our model and the val set for ablation studies, and finally report the results on test-dev set for comparison. In our experiments, only bounding box level annotations consider three types of backbones: ResNet-50 (He et al. (016)) ResNet-101 (He et al.] 2016) and 101-DXX-111DCCN22000 study.
used. We
Zhu & Dai 2019). For efficieccy, ResNet-50 is used in our ablation
4.2 IMPLEMENTATION DETAILS
Following the recent state-of-the-art object detectors such as FCOS (Tian et al. 2019) and Reti- naNet (Lin et al. 2017b), we adopt the ResNet (He et al. 2016) and ResNeXt (Xie et al. 2017) CNN network as our backbone architecture. ResNet and ResNeR are two fully convolutional networks, which are composed of a sequence of residual modules and were first used for image classification. Residual module first encodes the input feature by sequence of convolution and normalization layers, and then aggregates the generated feature map with the original input features. In order to predict objects with large scale variance, we also apply the Feature Pyramid Network (FPN) (Lin cet al.] (017a) in our approach, which combines the shallow layer features with deep layer features by the latent connection. To learn scale-robust detector, each level of FPN is responsible for certain scale of objects, making it very suitable for object detection. Specifically, we use FPN levels to make prediction, with stride 8, 16, 32, 64 and 128 compared with the original image, and each of the level is responsible for a certain scale of the objects: (0, 64), (64, 128]. (128, 256], (256, 512] and (512, INFI. We adopt ResNet-50 (He et al. 2016), ResNet-101 (He et al., 2016) and DCN2-ResNeXt-100 experiments.
(Xizhou Zhu & Dai 2019 as our backbone architecture in our
We train the model from weights pre-trained on ImageNet classification task and other parameters initialized by the same methods RetinaNet (Lin et al.) 20176). The model is trained with SGD optimization methods with 180k iterations with 16 images per mini-batch. The initial learning rate is set to 1e-2 and is reduced 10 times at 120k and 160k iterations. We re-scale the input images into 800x pixels before training. We use the same data augmentation strategy presented in (Tian) al. 2019) when training the model, and for each image, the top-100 predictions are produced.
4.3 EXPERIMENTAL RESULTS
Table I shows the results on the COCO val set by comparing our method with other popular anchor- free detectors mostly with ResNet-101 101 (for ComerNet and CenterNet we add results on Hourglass- 104 since they are designed based on Hourglass backbones). For comparison purposes, we also include two anchor-based detectors, including the popular RetinaNet (Lin et al. 2017b), TSD (Song et al.. 2020) and the state-of-the-at ATSS (Zhang et al. 2020b). We also include the new proposed detector DETR, which is based on the transformer Vaswani et al. 2017) and shows highly competitive results.
From the results, we can see that all mnchor-free/keeponnt-aass which uses predefined anchors and IoU matching methods. This confirms the advantage of keypoint- based detection methods over heuristic anchor- based designs. However, the existing anchor-free detectors worse than ATSS, which is recent ofate-of-the-an anchor-based method by borrowing and adapting some advanced strategies from anchor-free methods. By examining the results of our PolarNet, found that it outperforms all the existing keypoint- based detectors. Compared with Keypoint Position methods such as CornerNet, our model is able to capture more contextual information of the objects. Compared with other Keypoint Offsets methods such FCOS, our method represents keypoints by polar coordinates which eliminate poor keypoints and learn box via corner supervision. This merit significantly boosts the performance of our method. Finally, our supervision
methods outperform RetinaNet
method is even also better than the state-of-the- -ott-e-rrtt-----ee anchor-based ATSS, but eliminates the need of manually designed anchors. DETR the new proposed detection algorithm based on transformer which automatically learns the attention between each position of the image, and it predicts objects detection algorithm shows the
via a sequence of query position embedding. The transformer- great power to handle vision problems, but our method is still better than the DETR.
7

---

Published as a conference paper at ICLR 2021
Object Detectors
Anchor-fre free
Backbone
AP
AP
APTБ
APs
AP
APr.
RetinaNet (Lin
2017b
Anchor-based
R-101
39.1
59.1
42.3
21.8
42.7
50.2
TSD [Song aL. 2020
Anchor-bused
R-101
42.3
63.1
45.9
25.1
46.3
$6.5
ATSS Zhang
2020b
Anchor-bused
R-101
43.5
62.1
47.4
26.1
47.0
53.6
CoreeNet Law Deng
2018
Keypoint Position
R-101
44.1
32.0
13.3
33.3
42.7
CornerNet
Deng
2018
Keypoint Position
HG-104
40.5
56.5
43.1
19.4
42.7
53.9
RPDet Yang et al.
Keypoint Position
R-101
41.0
62.9
44.3
23.6
44.1
51.7
CenterNet (Zhou ct al.
2019u
Keypoint Offsets
R-101
53.0
36.9
CenterNet (Zhou ct al.
2019a
Keypuint Offsets
HG-104
42.1
61.1
45.9
24.1
45.5
52.8
FSAF (Zhu al. 2019
Keypoint Offsets
R-101
61.5
44.0
24.0
51.3
FoveaBox (Kong al
2020
Keypoint Offsets
R-101
60.1
43.5
23.3
45.2
54.5
FCOS (Tian et 20199
Keypoint Offsets
R-101
41.5
60.7
45.0
24.4
44.8
51.6
DETR (Carion et al. 20220
R-101
43.5
63.8
46.4
21.9
48.0
61.8
PolarNet (ours)
Keypoint Offsets
R-101
43.9
62.6
47.5
26.7
47.5
56.8
Table Performance evaluation of popular keypoint based detectors and two anchor-baesd detectors. The models are trained on MSCOCO train with 1 15k images, and they are validated on MSCOCO val set with 5k images. "R-101" denotes ResNet-101 backbone and "HG-104" denotes Hourglass- 104 backbone. For the backbone architecture, ResNet-101 (R-101) is pre-trained on ImageNet, while Hourglass network (HR-104) is trained from scratch.
4.4 ABLATION STUDY
This ablation study aims to examine the effectiveness of polar coordinates and corner learning in our PolarNet. T shows the results of our ablation study.
We first examine the effectiveness of polar coordinates. We set vanilla FCOS as baseline and compare the baseline with FCOS with polar coordinates (we denote this FCOS variant as FCOS-Polas Notably, we only change the encoding method of vanilla FCOS with Polar coordinates, without any additional computation cost. In vanilla FCOS, the gradient of the localization error each keypoint suppressed by its corresponding centerness score, reduce the impact of large-scale variance. And thus we first train the vanilla FCOS and FCOS-Polar without the centerness. From the results in Table 2 FCOS cannot converge if trained without the centerness, while our FCOS-Pol can still achieves highly competitive results (37.4%). which even comparable with the FCOS trained with centerness score (37.6%). This experiment indicates the effectiveness of the polar coordinates to handle scale issue. Then we train FCOS-Pola with centerness score, and this variant can still outperform vanilla FCOS (38.0% vs 37.6%) Finally, we train our FCOS-P Poolr based on the AutoSample strategy, and this variant achieves 38.9% AP on COCO metric, which even outperforms the best FCOS variant (38.5%) without any additional feature enhancement.
We then explore the impact of corner features. We integrate corner features with vanilla FCOS (denoted as (ECOS-Corner) and this variant achieves 38.4% on COCO metric, which significantly improves the accuracy (38.4% vs 37.6%), indicating the effectiveness of corner learning module. Finally, we learn our PolarNet and it achieves 39.6% AP on COCO metric, which is significantly better than the best FCOS variant and it is also highly competitive result among all anchor-free detection methods.
Methods
AP
APsa
AP7s
APs
AРM M
APL
FCOS w/o Centerness Tian et al. 2019)
fail to converge
FCOS w/ Centerness TTan et al. 2019
37.6
56.1
40.8
22.1
41.4
48.8
FCOS-Polat w/o Centerness
37.4
56.3
40.0
22.0
41.2
48.0
FCOS-Polar w/ Centerness
38.0
56.7
40.8
22.3
41.7
49.2
FCOS-Polar w/ AutoSample
38.9
57.5
41.8
22.4
43.0
FCOS-Corerr w/ Centerness
38.4
56.8
41.1
22.9
42.4
49.5
FCOS w/ imprv
Tian al. 2019)
38.5
57.4
41.4
22.3
42.5
49.8
PolarNet
39.6
57.9
42.9
23.2
43.5
51.8
Table 2: Ablation study different components in PolarNet. Models are trained COCO train2017 and tested on COCO va12017 with ResNet-50.
8

---

Published as a conference paper at ICLR 2021
4.5 COMPARISON WITH STATE-OF-THE ORT DETECTORS
We compare PolarNet with other state-of-the-ar detectors on COCO test-dev set. Unlike the previous experiments, train our models backbone ResNeXt-1OI-DCNNV. Table shows the results on COCO test-dev under the single-model settings, and the inference time is based on Titan-Xp. PolarNet outperforms all single-stagg stage detectors by substantial margin (except ATSS), also outperform variety of two-stage/multi-staag detectors, and it achieves the best results among all the keypoints based detectors. This promising result validates our hypothesis of learning keypoints based on polar coordinates with corner supervision is able to better predict the objects. Specifically, compared with other center-based methods such as CenterNet or FCOS, our method not only extracts features from the central region, but also encodes features from other informative areas based the shape of the objects. Besides, our comner supervision
scale-invarian and thus shows better results localization. Finally, compared with the SOTA anchor-base ATSS, method eliminates the need of using anchors and thus avoids heuristic anchor design.
Methou
Backbone
FPS
AP
AP5n
AP75
APs
AP
APr.
Anchor-based
Multi-stage
FRCN-FPN
Lin
2017a
R-101
36.2
59.1
39.0
18.2
39.0
48.2
Finess-NMS
Tycheen-mmih Fetensar
2015
R-101
5.0
41.8
60.9
44.9
21.5
45.0
$7.5
Cascade R-CNN Car
2018
R-101
42.8
62.1
46.3
45.5
55.2
Libra R-CNN Pann ot al 2019
X-101-64x4d
5.6
43.0
64.0
47.0
25.3
45.6
54.6
FreeAnchor Zhang
X-101-32x8d
TridentNet èt 2019
2019)
R- R-101-DCN
1.3
46.8
67.6
51.5
28.0
51.2
60.5
Dynamic R-CNN Ziang
2020u
R-101
42.0
60.7
45.9
22.7
44.3
54.3
Cascale CNN WABH.
Wang
2020
R 101
43.3
60.9
46.2
23.8
46.5
55.7
TSD++ Song et al, 2020
X-101-64x4d-DCC
49.2
70.1
53.8
33.2
53.1
63.7
Single-stage
RetinaNet (Lin al
20176
R-101
8.0
39.1
$9.1
42.3
21.8
42.7
50,2
40.9
AB-ESAF Zhu
2019
X-101-64x4d
4.2
42.9
63.8
46.3
26.6
46.2
52.7
M2Dei Zhha el
2019
VGG 16
11.8
41.0
59.7
45.0
22.1
46.5
53,8
ATSS (Thang
20206
X-10I 1101-64x4d-DCN
7.1
47.7
66.5
51.9
29.7
50.8
59.4
Anchor-free
GA-KetinaNet (Wane al.
2019
R-50
10.8
37.1
56.9
40.0
20.1
40.1
48.0
GA-FRCN Wane
2019
R-50
9.4
39.8
59.2
433.5
21.8
42.6
50.7
ExtremeNet Zhou &
20196
HG-104
2.8
40.2
55.5
43.2
20.4
43.2
53.1
ConeerNet Law
53.9
FCOS Tiah el al. 2019)
R-101
9.3
41.5
60.7
45.0
24.4
44.8
$1.6
CenterNet Zhop
ConterNet Düan el
2019
HG-104
3,3
44.9
62.4
48.1
25.6
47.4
57.4
RPDet Yang al. 2019)
R- 101-DDN
45.0
66.1
49.0
26.6
48.6
57.5
FCOS
2019
X-10I 64x4d DCN
4.7
46.5
65.7
S0.6
28.9
49.2
58.1
FoveaBox Kong et 2024
R-101
11.2
40.6
60.1
43.5
23.3
45.2
54.5
FoveaBux (Kong
A144
SAPD Zhi m. 2120
R 101
11.2
43.5
63.6
46.5
24.9
46.8
$4.6
Zhir
20.00
X-101-64x48DDCN
47.4
51.1
28.1
50.3
61.5
ContripetalNat Dong èt
2020
HG-104
45.8
63.0
49,3
25.0
48.2
58.7
CentripetalNett- Dong al
20200
HG-104
47.8
65.0
51.5
28.9
50.2
59.4
CPN (Dun
20200
IIG-104
47.0
65.0
$1.0
26.5
50.2
60.7
CPN Duan et al.
2020
HG-104
49,2
67.4
53.7
31.0
51.9
62.4
PolarNet (ours)
PolarNet (ours)
X-10I 64x4d.DCN
4.3
47.8
67.0
52.1
29.7
50.9
59.5
PolarNote+ (ours)
X-101-64xA6-DDCN
50.3
69.0
55.4
33.5
53.5
61.5
Table 3: Comparison of PolarNet with other state-of-the-ar two-stage one-stage detectors (single- model results). All models were trained on COCO train set and tested on test-dev set. The "++" means the improved inference version of the proposed algorithm, i.e., inference with multi-scale and aspect-ratios, which is widely used literature.
CONCLUSION
In this paper, we presented unified view of popular anchor-free object detectors from the object modeling based detection perspective. We argue that the existing keypoint based detectors often yield a lot poor quality keypoints based on Cartesian coordinates and the learning detector is also based on scale-variant features. To overcome the limitation, based detector by representing keypoints using polar coordinates which avoids poor quality keypoints well learning scale-inn invariant corner features for object localization. Our new detector obtained the highly competitive results on the MS COCO test-dev set under the single-mon model single-scale and multi-scale settings.
proposed PolarNet, new keypoint

---

Published as a conference paper at ICLR 2021
REFERENCES
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, 2018
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Yihong Chen, Zheng Zhang, Yue Cao, Liwei Wang, Stephen Lin, and Han Hu. Reppoints v2: Verification meets regression for object detection. In NewrlPS, 2020.
Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fen: Object detection via region-based fully convolutional networks. In NewrIPS, 2016.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
Zhiwei Dong, Guoxuan Li, Yue Liao, Fei Wang, Pengju Ren, and Chen Qian. Centripetalnet: Pursuing high-quality keypoint pairs for object detection. In CVPR, 2020.
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi. Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. ICCV, 2019.
Kaiwen Duan, Lingxi Xie, Honggang Qi, Song Bai, Qingming Huang, and Qi Tian. Corner proposal network for anchor-free, two-stage object detection. In ECCV, 2020.
Ross Girshick. Fast r-cnn. ICCV, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Tao Kong, Fuchun Sun, Huaping Liu, Yuning Jiang, Lei Li, and Jianbo Shi. Foveabox: Beyond anchor-based object detector. IEEE Transactions on Image Processing, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In NeurlPS, 2012.
Hei Law and Jia Deng. Comernet: Detecting objects as paired keypoints. In ECCV, 2018.
Hei Law, Yun Teng, Olga Russakovsky, and Jia Deng. Comernet-lite: Efficient keypoint based object detection. arXiv preprint 19040000000 2019.
Yanghao Li, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Scale-aware trident networks for object detection. In ICCV, 2019.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017a.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017b.
Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen. Deep learning for generic object detection: A survey. International journal of computer vision, 128(2):261-318, 2020.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. SSD: Single shot multibox detector. ECCV, 2016.
Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dabua Lin. Libra r-cnn: Towards balanced learning for object detection. In CVPR, 2019.
10

---

Published as a conference paper at ICLR 2021
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, lan Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In CVPR, 2019.
Guanglu Song, Yu Liu, and Xiaogang Wang. Revisiting the sibling head in object detector. CVPR, 2020.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In ICCV, 2019.
Lachlan Tychsen- Tychsen-Smith Smith and Lars Petersson. Improving object localization with fitness nms and bounded iou loss. In CVPR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Jiaqi Wang, Kai Chen, Shuo Yang, Chen Change Loy, and Dahua Lin. Region proposal by guided anchoring. In CVPR, 2019.
Jiaqi Wang, Wenwei Zhang, Yuhang Cao, Kai Chen, Jiangmiao Pang, Tao Gong, Jianping Shi, Chen Change Loy, and Dahua Lin. Side-aware boundary localization for more precise object detection. In ECCV, 2020.
Xiongwei Wu, Doyen Sahoo, and Steven CH Hoi. Recent advances in deep learning for object detection. Neurocomputing. 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
Stephen Lin Xizhou Zhu, Han Hu and Jifeng Dai. Deformable convnets v2: More deformable, better results. In CVPR, 2019.
Ze Yang, Shaohui Liu, Han Hu, Liwei Wang, and Stephen Lin. Reppoints: Point set representation for object detection. In ICCV, 2019.
Hongkai Zhang, Hong Chang, Bingpeng Ma, Naiyan Wang, and Xilin Chen. Dynamic R-CNN: Towards high quality object detection via dynamic training. In ECCV, 2020a.
Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei, and Stan Z Li. Single-shot refinement neural network for object detection. In CVPR. 2018.
Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li. Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In CVPR, 2020b.
Xiaosong Zhang, Fang Wan, Chang Liu, Rongrong Ji, and Qixiang Ye. FreeAnchor: Learning to match anchors for visual object detection. In NewrlPs, 2019.
Qijie Zhao, Tao Sheng, Yongtao Wang, Zhi Tang, Ying Chen, Ling Cai, and Haibin Ling. M2det: single-shot object detector based on multi-level feature pyramid network. In AAAI, 2019.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. In arXiv preprint arXiv:1904.07835 07855,, 2019a.
Xingyi Zhou, Jiacheng Zhuo and Philipp Krahenbuhl. Bottom-up object detection by grouping extreme and center points. In CVPR, 2019b.
Chenchen Zhu, Yihui He, and Marios Savvides. Feature selective anchor-free module for single-shot object detection. In CVPR, 2019.
Chenchen Zhu, Fangyi Chen, Zhiqiang Shen, and Marios Savvides. Soft anchor-point object detection. In ECCV, 2020.
11

---

Published as a conference paper at ICLR 2021
APPENDIX
.1 ANALYSIS ON KEYPOINTS SELECTION
From Figure 2 although polar system has significantly reduced the number of poor quality keypoints, we can still further improve the detection algorithms. this section, we aim explore the strategy to further reduce the impact of these poor quality keypoints. We have three keypoint selection strategies shown in Figure
FCOS-Polar w/ dual-head (Figure 4 Ha)). Two polar heads are learned in FCOS-Polat which point to different corner pairs, and the variance of learned offsets are also learned. During inference, the head with smaller offsets variance will be selected.
FCOS-Polas w/ CemterSample (Figure Vanilla center sampling strategy adopted in FCOS (Tian et al., 2019), where only the small central regions the objects will be used which determined by the FPN stride.
FCOS-Polar w/ AutoSample (Figure We adopt a shape-awar aware sampling strategy which samples keypoints with sufficient centerness score (we set threshold as 7)).
From Table all variants show some improvement compared with the baseline, and we found FCOS-Polar w/ AutoSample achieves best results. Compared with the other two variants, autosample a parameter-free strategy and is able to learn high quality keypoints based on the shape of the objects. The central region of vanilla center sampling is determined by the FPN stride, which has limited relationship with the shape of different objects, and thus fails to capture sufficient contexts of the objects. While AutoSample not only reduces the poor quality keypoints but also keep sufficient training samples based on the shape of different objects.
(a) FCOS Poolar w/ dual-head
(b) FCOS-Polaa CenterSample
(c) FCOS-Polar AutoSample
Figure 4: (a), (b) and (c) show the comparison of three keypoints selection strategies. The model is trained on COCO train2017 and tested on COCO val2017. All the models are based on ResNet-50 without corner feature enhancement.
We further conduct the ablation study to explore the impact of 7 in AutoSample. The results are shown Table 5 and the heatmaps of different y are shown Figure From the heatmaps, the variants of AutoSample show good trade-off between rejecting poor quality keypoints and keeping sufficient information compared with vanilla center sampling, and they also obtain significant improvement. Based on the results, we select 0.4 in all our experiments.
Methods
AP
AP
50
AP75
AP$ S
AP M
AP, I
FCOS-Polar
38.0
56.7
40.8
22.3
41.7
49.2
FCOS-Polar w/ dual-head
38.3
57.0
41.4
22.4
42.1
49.4
FCOS-Polar w/ CenterSample
38.6
57.1
42.0
22.6
42.2
50.2
FCOS-Polar AutoSample
38.9
57.5
41.8
22.4
43.0
50.5
Table Ablation study on different keypoints sampling strategy in PolarNet. Models are trained on COCO train2017 and tested on COCO va12017 with ResNet-50.
12

---

Viy denote the confidence score and binary class label of position (i.j). And O₃ and 6₃ denote the learned and target offset of corner & from down sampled feature map to the original image. After the Published as a conference paper at ICLR 2021
Value of 7
AP
50 50
AP
APs
AРM
APL
0.0
38.0
56.7
40.8
22.3
41.7
49.2
0.2
38.2
56.9
41.1
22.4
42.0
49.4
0.3
38.7
57.4
41.7
21.8
42.7
50.3
0.4
38.9
57.5
41.8
22.4
43.0
50.5
0.5
38.3
56.4
41.1
21.6
42.4
50.2
0.6
37.8
55.9
40.6
22.0
41.6
50.0
Table 5: Ablation study on different values of y AutoSample. Models are trained on COCO train2017 and tested on COCO val2017 with ResNer-50.
.2 CORNER FEATURE LEARNING
Corner features are scale-invariant, and thus are particularly suitable enhance the feature represen- tation of our PolarNet. In addition, the corner features can further enhance the feature representation around the corners where keypoints around are difficult to optimize. We follow the similar training strategy as CornerNet (Law & Deng, 2018) to train the corner features as:
ΣΣ
H
W
(1 - Pij) log (Piz) if Vij = 1 yuy) (pt)) log Pij)
Leomer-els
N
(1
otherwise
1
Leomer-loc
I (Ok, Ōτ)
Leomer
Leorner-lls
Leooner-loo
(9)
where N, H and W denote the corner number, height and width of the feature map. while Pıy and
features are learned, integrate the learned scale-invariant corner features into the original PolarNet features to make it scale-insensitive. In our final model, foreground features are also used to further improve the results.
23
(a) CenterSample
(b) AutoSample(y = 0.2)
(c) AutoSample(y = 0.3)
4
1
(d) AutoSample(y 0.4)
AutoSample(? 0.5)
(f) AutoSample(y 0.6)
Figure 5: (a) shows the heatmap of CenteSample, and (b)-(f) show the heatmaps of AutoSample based on different 7,
13


Reviews:
[{"review_id": "yrRj5NFcelM", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new key-point based object detector, PolarNet, which predicts the distances between key-points and corner pairs (such as top-left and bottom-right pair or top-right and bottom-left pair) on polar coordinates. This is different from other key-point based object detectors such as FCOS which predicts distances between key-points and bounding box boundaries on cartesian coordinates.  The authors claim that the advantage of representing the offsets in the form of polar coordinates is this representation reduces the variance in the offsets, which makes learning easier.\n\nPros:\nThis is an interesting approach and new, to the best of my knowledge, in the context of object detection. The authors show that PolarNet outperforms other approaches such as FCOS and FoveaBox under the same backbone network, ResNet-101. The use of polar coordinates improves the performance of FCOS by more than 4% which shows the effectiveness of the polar coordinates. With a larger backbone and deformable convolution, PolarNet demonstrates state-of-the-art performance among all anchor-free detectors on the challenging COCO dataset.\n\nCons:\nI am confused about the corner supervision in section 3.4.2. It seems to me that the corner supervision is to train PolarNet to predict the offsets which are in polar coordinates. But the authors also apply this to FCOS (FCOS + Centerness + Corner) and compare it with FCOS with polar coordinates (FCOS + Polar).  I don’t understand the difference between them. How do the authors apply corner supervision if FCOS is predicting on cartesian coordinates (i.e. FCOS + Centerness + Corner)? How is that different from FCOS + Polar exactly? Or do I misunderstand the meaning of corner supervision? The authors also mention in the section where they introduce corner supervision that they train a regressor based on corner features. What does corner supervision mean exactly? Does it mean the authors extract features from the corners and use the features together with the key-point features when they predict the offsets? Or do they simply refer to the regression loss function?\n\nThe use of IoU loss seems to a bit redundant. It seems that the $L_{corner}$ already trains the network to predict the offsets. Why do the authors still need the IoU loss? The authors should provide an ablation study to demonstrate how the IoU loss is affecting the performance of PolarNet.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ilc7fO0jFSx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nIn this paper, the author proposes a new anchor-free keypoint based detector, which learns keypoints based on polar coordinates. It can avoid the large variance of learned offsets compared to the existing anchor-free detectors and make bounding box prediction scale-invariant.\n\nReasons for score:\n\nOverall, I vote for accepting. My major concern is about the clarity of the paper and some additional analysis (see cons below). Hopefully the authors can address my concern in the rebuttal period.\n\nPros:\n\n1.This paper proposes a new keypoint based object detector, which represents keypoints based on polar coordinates. It can avoid poor quality keypoints suffered by the existing keypoint based detectors and make regression scale-invariant.\n\n2.Experiments are well thought out and highlight the key advantages of the method over other keypoint based detectors. \n\nCons:\n\n1.In the Section 3.4.2, why use tangent function for angle regression? For tan, its derivative is $\\sec^{2}$. When $|\\theta_{br}-\\theta_{br}^{*}| \\to \\frac{\\pi}{2}$, the gradient is very large. Will this condition happen? In addition, are there some constraints or operations on $\\theta$ to make $\\theta \\in (0, \\frac{\\pi}{2})$?\n\n2.For the training, IoU threshold is adaptively change. Why use this trick? It may lead to unfair comparison with other methods. Could you provide some experiments with fixed IoU threshold?\n\n3.IoU loss is kept in the training. Why keep it? Could you provide some experiments without it to demonstrate its benefits?\n\nQuestions during rebuttal period:\n\nPlease address and clarify the cons above\n\nSome typos:\n\n(1)3.1: $\\theta \\in \\begin{Bmatrix} 0, & \\frac{\\pi}{2}  \\end{Bmatrix}  \\to \\theta \\in (0, \\frac{\\pi}{2})$\n\n(2)4.1 5th line: four types -> three types ", "extra_scores": {}, "extra_text": {}}, {"review_id": "V4JzA1A6vB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper proposes an anchor-free object detector that does bounding box regression in the polar coordinate instead of in the Cartesian coordinate. The motivation of doing this is because there are larger variance in offset vectors in the Cartesian coordinate (the extreme case when a point is on one of the four corners of the bounding box, the offset vector becomes [0, w, 0, h]). The authors propose a solution to regress to the pair of corners (either TL+BR or TR+BL) in the polar coordinate, and select the corner pair that gives the smallest variance during training.\n\n##########################################################################\n\nPros: \n\nExperiment results show using polar coordinate is effective.\n\n##########################################################################\n\nCons:\n\n1. The authors did some analysis on the variance of the offset vector in Section 3.1, however, I think the analysis is not enough. First, the analysis only contains the worst case analysis, that is, the range of offset targets. And the author directly concluded from this: PolarNet \"significantly reduces the variance\" (page 4 last line). What is the ratio of variance under cartesian and polar coordinate to make it \"significant\"? I do not see any number either theoretically proves it or empirical analysis of the variance during training.\n\n2. The term \"keypoint\" used in this paper is confusing. Sometimes the \"keypoint\" refers to corner points of the box (\"Keypoint Position\") and other times the \"keypoint\" simply refers to any point within the bounding box (\"Keypoint Offsets\").\n\n3. The introduction of PolarNet is not clearly presented, specifically there are several confusing points:\n- Section 3.4.1, what is the usage of t_{x,y}? I don't see how t is used during inference.\n- Section 3.5, \"we select the optimal box from b_{x,y} as the final output of the predicted box\", what is \"optimal box\"?\n- Figure 3, why the \"corner supervision\" comes from the feature map? I don't see how corner supervision uses any feature.\n- The proposed corner supervision is simply the L1 loss, and there are methods that already use it with IoU loss. I don't think it is a contribution and section 3.4.2 and 3.4.3 should be combined.\n\n4. Experiments are not solid:\n- There are ways to reduce variance of offsets under cartesian coordinate, e.g. only use points within the center region of the bounding box to learn offset. Such experiment should be compared.\n- The importance of extra loss function is also not studied, what are the benefit of using more losses?\n- From reading Section 3, I feel the method is exactly as applying FCOS + polar coordinate, but Table 2 shows there is still some gap. Where does the extra gain come from?\n- I checked the FCOS paper and found the R101 results in the paper is 43.2 but the number in this paper is 41.5.\n\n##########################################################################\n\nReasons for score: \n\nOverall, I vote for rejecting. This paper proposed an interesting idea, but I think way it is presented is not good enough to be accepted. Specifically, I think the paper still misses analysis on the variance of offset prediction, and also misses some important ablation studies. Furthermore, the paper is not well-written and requires some revision.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FZ70y38XOH_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nIn this paper, the authors propose a simple but effective keypoint-based anchor-free object detection system. The main idea is to replace the Cartesian coordinate with the polar coordinate, compared to the closest related work, FCOS. According to the extensive empirical results, the proposed system achieves a better trade-off between speed and accuracy. \n\nThe overall writing looks good to me. The storyline is consistent and well-motivated. The authors provide enough detail to shed light on the design choices for the state-of-the-art anchor-free detector. The figures and tables are also quite informative. For example, I particularly love the figure 1 because it helps the readers catch up with the most recent progress on keypoint-based object detection frontier. It could be better if the authors could describe more details in the caption. By the way, the black color of rho and theta should be changed into a lighter one.\n\n# Questions\n\n1. scale-sensitive vs. scale-invariant\nIn this paper, the author mentioned the scale-related terminology many times. I wonder if the authors could explain more detail about why PolarNet is scale-invariant? From my perspective, the offset regression is scale-sensitive because the target numbers are strongly related to the actual object size. Even though the proposed method utilizes the corner points to localize objects, the bounding box offset part is still scale-sensitive, right?\n\n2. center-based method + polar coordinate\nI wonder if the authors could try to put the polar coordinate offset regression into a center-based anchor-free detector. For example,  CenterNet[1] regresses the bounding box offset in the Cartesian coordinate system. What if we change the distance encoding to polar coordinate? In section 4.4, the authors claim, \"Speciﬁcally, compared with other center-based methods such as CenterNet or FSFA, our method not only extracts features from the central region, but also encodes features from the whole bounding boxes.\" I wonder how much performance gain could be seen if we only change the coordinate system. It could be a helpful ablation study to support this claim.\n\n# Reference\n[1] Zhou, Xingyi, Dequan Wang, and Philipp Krähenbühl. \"Objects as points.\" arXiv preprint arXiv:1904.07850 (2019).\n\n---- Post-rebuttal comments----\n\nThe rebuttal and the paper revision address my concerns. I keep my original rating.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received overall positive scores. One reviewer (R3) recommended clear reject. 

All reviewers agree that the paper introduces a novel idea and its effectiveness is supported by the experimental results. There are concerns about clarity of presentation and certain missing analyses, which have been addressed by the authors in the rebuttal. Thus the ACs recommend acceptance.  
Decision: Accept (Poster)