# A Benchmark for the Evaluation of Corner Detectors
## 2 Benchmark Construction
### Benchmark Evaluation Metrics  
In previous studies, corner detectors were evaluated by using the precision [21; 43], recall [43], localization error (Le) [22; 25; 28; 37], and repeatability metrics [22; 25; 26; 27; 28; 37]. Among these metrics, the repeatability metric could be the most popular one used in the evaluation of corner detectors, which is based on the number of repeated corners between the original images and transformed images. Unfortunately, its evaluation mechanism has an obvious flaw: if a repeated corner is a false positive, then it will not make sense in the evaluation of corner detection performance. We denote the corner sets detected from the original and transformed images as \(\mathbf{C}_{o}\) and \(\mathbf{C}_{t}\), respectively. Let \(\mathbf{C}_{rep}=\mathbf{C}_{o}\cap\mathbf{C}_{t}\), which is the set of corners that have been repeatedly detected on the two images. Let \(N_{o}\), \(N_{t}\), and \(N_{rep}\) be the numbers of corners in the sets \(\mathbf{C}_{o}\), \(\mathbf{C}_{t}\), and \(\mathbf{C}_{rep}\), respectively. The existing repeatability metric REP is given by  
\[REP=\frac{N_{rep}}{2}(\frac{1}{N_{o}}+\frac{1}{N_{t}}). \tag{1}\]  
In this paper, we denote the set of ground-truth corners as \(\mathbf{C}_{gt}\) and let \(\mathbf{C}_{rgt}=\mathbf{C}_{rep}\cap\mathbf{C}_{gt}\), which is the set of ground-truth corners that have been repeatedly detected. Let \(N_{rgt}\) be the number of corners in the set \(\mathbf{C}_{rgt}\). Our proposed repeatability metric, (i.e., the _repeatability referenced to ground truth_ (RGT)), is then defined as  
\[RGT=\frac{N_{rgt}}{2}(\frac{1}{N_{o}}+\frac{1}{N_{t}}). \tag{2}\]  
For ease of understanding, Figure 3 demonstrates an example of comparing the flawed repeatability metric (REP) and our proposed repeatability metric (RGT). The corners detected in the original image (\(N_{o}=3\)) are shown in Figure 3a. The corners detected in a noise-corrupted version of the image (\(N_{t}=5\)) are presented in Figure 3b. According to Figure 3a,b, three corners were repeatedly detected in the two images (marked with black circles) in Figure 3c (i.e., \(N_{rep}=3\), and thus \(REP=0.8\)). Only one ground-truth corner was repeatedly detected in the two images (marked with a black circle) in Figure 3d (i.e., \(N_{rgt}=3\)). As a consequence, we found that \(RGT=0.2667\). Figure 2: An example for labeling ground truth. (**a**) The original image. (**b**) The result of line segment detection. (**c**) Manually approved line segments. (**d**) The intersections of of line segments. (**e**) Corner annotations. Due to the lack of corner annotations, the precision, recall and F score are rarely used to evaluate corner detection performance. Most previous studies only use the repeatability and localization error, which are unfortunately not authoritative or unified. In this paper, a set of benchmark evaluation metrics is suggested, including the precision, recall, APR (the arithmetic mean of precision and recall), F score, Le and our newly developed RGT. Note that all our suggested evaluation metrics are referenced to ground truths, where a fundamental consideration is that only the repeated detection of true positives can be taken as a merit of the used algorithm. Our benchmark evaluation metrics are as follows:  
(1) Precision is expressed as  
\[P=\frac{N_{r}}{N_{r}+N_{f}}, \tag{3}\]  
where \(N_{r}\) denotes the number of corners in the set \(\mathbf{C}_{r}=\mathbf{C}_{o}\cap\mathbf{C}_{gt}\) and \(N_{f}\) is the number of falsely detected corners (i.e., false positives). (2) The recall is expressed as  
\[R=\frac{N_{r}}{N_{r}+N_{m}}, \tag{4}\]  
where \(N_{m}\) is the number of undetected corners (i.e., false negatives). (3) The APR is expressed as  
\[APR=\frac{P+R}{2}. \tag{5}\]  
(4) The F score is expressed as  
\[\text{F-Score}=\frac{(1+\beta^{2})PR}{\beta^{2}(P+R)}, \tag{6}\]  
where \(\beta\) is assigned to be 1.