SlowGuess/NeurIPS_2023_OCR huggingface dataset:
FAIR1M [66]: It contains more than 1 million instances and more than 40,000 images for fine-grained object recognition in high-resolution remote sensing imagery. The dataset is annotated with five categories and 37 fine-grained subcategories. We split the images into  $1,024 \times 1,024$  patches with an overlap of 200 pixels and a scale rate of 1.5 and merge the results for testing. The performance is evaluated on the FAIR1M-1.0 server.
Experimental settings: We adopt the FCOS [26] detector with ResNet50 [58] backbone and FPN [59] neck as the baseline method, based on which we develop our H2RBox-v2. We choose average precision (AP) as the primary metric to compare with existing literature. For a fair comparison, all the listed models are configured based on ResNet50 [58] backbone and trained on NVIDIA RTX3090/4090 GPUs. All models are trained with AdamW [67], with an initial learning rate of 5e-5 and a mini-batch size of 2. Besides, we adopt a learning rate warm-up for 500 iterations, and the learning rate is divided by ten at each decay step. "1x", "3x", and "6x" schedules indicate 12, 36, and 72 epochs for training. "MS" and "RR" denote multi-scale technique [62] and random rotation augmentation. Unless otherwise specified, "6x" is used for HRSC and "1x" for the other datasets, while random flipping is the only augmentation that is always adopted by default.
Table 2: Results on the DOTA-v1.0 dataset.
Method	Sched.	MS	RR	Size	FPS	\(AP_{50}\)
RBox-supervised	RepPoints (2019) [41]	1x	1,024	24.5	68.45
RetinaNet (2017) [25]	1x	1,024	25.4	68.69
KLD (2021) [38]	1x	1,024	25.4	71.24
KFIoU (2023) [40]	1x	1,024	25.4	71.61
GWD (2021) [37]	1x	1,024	25.4	71.66
PSC (2023) [36]	1x	1,024	25.4	71.92
SASM (2022) [63]	1x	1,024	24.4	72.30
\(R^3\text{Det}\)(2021) [30]	1x	1,024	20.0	73.12
CFA (2021) [64]	1x	1,024	24.5	73.84
Oriented RepPoints (2022) [43]	1x	1,024	24.5	75.26
\(S^2A\text{-}\text{Net}\)(2022) [31]	1x	1,024	23.3	75.81
FCOS (2019) [26]	1x	1,024	29.5	72.44
FCOS (2019) [26]	3x	✓	1,024	29.5	74.75
FCOS (2019) [26]	1x	✓	✓	1,024	29.5	77.68
HBox-supervised	BoxInst-RBox (2021) [21]1	1x	960	2.7	53.59
BoxLevelSet-RBox (2022) [22]2	1x	960	4.7	56.44
SAM-ViT-B-RBox (2023) [23]3	1x	1,024	1.7	63.94
H2RBox (FCOS-based) (2023) [20]4	1x	1,024	29.1	67.82
H2RBox (FCOS-based) (2023) [20]5	1x	1,024	29.1	70.05
H2RBox (FCOS-based) (2023) [20]4	1x	✓	1,024	29.1	74.40
H2RBox (FCOS-based) (2023) [20]5	1x	✓	1,024	29.1	75.35
H2RBox-v2 (FCOS-based)	1x	960	31.6	71.46
H2RBox-v2 (FCOS-based)	1x	1,024	29.1	72.31
H2RBox-v2 (FCOS-based)	3x	✓	1,024	29.1	74.29
H2RBox-v2 (FCOS-based)	1x	✓	1,024	29.1	77.97
H2RBox-v2 (FCOS-based)	1x	✓	✓	1,024	29.1	78.25
H2RBox-v2 (FCOS-based, Swin-T)6	1x	✓	✓	1,024	24.0	79.39
H2RBox-v2 (FCOS-based, Swin-B)6	1x	✓	✓	1,024	12.4	80.61
1 "RBox" means the minimum rectangle operation is performed on the Mask to obtain RBox.
Evaluated on NVIDIA V100 GPU due to the excessive RAM usage.
3 The code is available at https://github.com/Li-Qingyun/sam-mmrotate.
Results reported in the original paper [20].
5 Results reproduced by us with same infrastructure for a fair comparison.
6 Using Swin Transformer [65] as backbone on four NVIDIA A100 GPUs with batch size  $= 4$
DOTA-v1.0: Table 2 shows that H2RBox-v2 outperforms HBox-Mask-Rbox methods in both accuracy and speed. Taking BoxLevelSet-RBox [22] as an example, H2RBox-v2 gives an accuracy of  $15.02\%$  higher, and a x7 speed faster by avoiding the time-consuming post-processing (i.e. minimum circumscribed rectangle operation). In particular, the recent foundation model for segmentation i.e. SAM [23] has shown strong zero-shot capabilities by training on the largest segmentation dataset to date. Thus, we use a trained horizontal FCOS detector to provide HBoxes into SAM as prompts, so that the corresponding masks can be generated by zero-shot, and finally the rotated RBoxes are obtained by performing the minimum circumscribed rectangle operation on the predicted Masks. Thanks to the powerful zero-shot capability, SAM-RBox based on ViT-B [68] in Table 2 has achieved  $63.94\%$ . However, it is also limited to the additional mask prediction step and the time-consuming post-processing, only 1.7 FPS during inference.
In comparison with the current state-of-the-art method H2RBox, to make it fair, we use the reproduced result of H2RBox, which achieves  $70.05\%$ ,  $2.23\%$  higher than the original paper [20]. In this fair comparison, our method outperforms H2RBox by  $2.26\%$  (72.31% vs. 70.05%, both w/o MS). When MS is applied, the improvement is  $2.62\%$  (75.35% vs. 77.97%, both w/ MS).
Furthermore, the performance gap between our method and the RBox-supervised FCOS baseline is only  $0.13\%$  (w/o MS and RR) and  $0.46\%$  (w/ RR). When MS and RR are both applied, our method outperforms RBox-supervised FCOS by  $0.57\%$  (78.25% vs. 77.68%), proving that supplemented with symmetry-aware learning, the weakly-supervised learning can achieve performance on a par with the fully-supervised one upon the same backbone neural network. Finally, H2RBox-v2 obtains  $80.61\%$  on DOTA-v1.0 by further utilizing a stronger backbone.
Table 3:  ${\mathrm{{AP}}}_{50}$  performance on the DOTA-v1.0/1.5/2.0,HRSC,and FAIR1M datasets.
Method	DOTA-v1.0	DOTA-v1.5	DOTA-v2.0	HRSC	FAIR1M
RetinaNet (2017) [25]	68.69	60.57	47.00	84.49	37.67
GWD (2021) [37]	71.66	63.27	48.87	86.67	39.11
S2A-Net (2022) [31]	75.81	66.53	52.39	90.10	42.44
FCOS (2019) [26]	72.44	64.53	51.77	88.99	41.25
Sun et al. (2021) [49]1	38.60	-	-	-	-
KCR (2023) [50]2	-	-	-	79.10	-
H2RBox (2023) [20]	70.05	61.70	48.68	7.03	35.94
H2RBox-v2	72.31	64.76	50.33	89.66	42.27
1 Sparse annotation for horizontal/vertical objects. The result is cited from their paper.
2 Transfer learning from DOTA (RBox) to HRSC (HBox). The result is cited from their paper.
Table 4: Ablation with different SS losses.
Dataset	PSC	\( \ell_s \)	AP	\( AP_{50} \)	\( AP_{75} \)
DOTA	24.24	52.24	19.48
✓	0.01	0.77	0.02
✓	10.49	27.57	6.15
✓	✓	40.69	72.31	39.49
HRSC	2.25	7.83	0.62
✓	48.95	88.52	50.03
✓	0.31	0.88	0.13
✓	✓	58.03	89.66	64.80
Table 5: Ablation with different WS losses.
Dataset	\( \ell_p \)	RR	AP	\( AP_{50} \)	\( AP_{75} \)
DOTA	39.35	71.49	37.03
✓	11.93	29.34	7.86
✓	40.69	72.31	39.49
✓	✓	40.17	71.79	39.77
HRSC	56.20	89.58	61.84
✓	41.10	87.19	33.97
✓	58.03	89.66	64.80
✓	✓	63.82	89.56	76.11
DOTA-v1.5/2.0: As extended versions of DOTA-v1.0, these two datasets are more challenging, while the results present a similar trend. Still, H2RBox-v2 shows considerable advantages over H2RBox, with an improvement of  $3.06\%$  on DOTA-v1.5 and  $1.65\%$  on DOTA-v2.0. The results on DOTA-v1.5/2.0, HRSC, and FAIR1M are shown in Table 3.
HRSC: H2RBox [20] can hardly learn angle information from small datasets like HRSC, resulting in deficient performance. Contrarily, H2RBox-v2 is good at this kind of task, giving a performance comparable to fully-supervised methods. Compared to KCR [50] that uses transfer learning from RBox-supervised DOTA to HBox-supervised HRSC, our method, merely using HBox-supervised HRSC, outperforms KCR by  $10.56\%$  (89.66% vs. 79.10%).
FAIR1M: This dataset contains a large number of planes, vehicles, and courts, which are more perfectly symmetric than objects like bridges and harbors in DOTA. This may explain the observation that H2RBox-v2, learning from symmetry, outperforms H2RBox by a more considerable margin of  $6.33\%$  (42.27% vs. 35.94%). In this case, H2RBox-v2 even performs superior to the fully-supervised FCOS that H2RBox-v2 is based on by  $1.02\%$  (42.27% vs. 41.25%).
Loss in SS branch: Table 4 studies the impact of using the snap loss (see Sec. 3.4) and the angle coder. Column "PSC" indicates using PSC angle coder [36] and "w/o PSC" means the conv layer directly outputs the angle. Column  $\ell_{s}$  with check mark denotes using snap loss (otherwise using smooth L1 loss). Without these two modules handling boundary discontinuity, we empirically find that the loss could fluctuate in a wide range, even failure in convergence (see the much lower results in Table 4). In comparison, when both PSC and snap loss are used, the training is stable.
Loss in WS branch: Table 5 shows that CircumIoU loss with random rotation can further improve the performance, which H2RBox is incapable of. “ $\ell_{\mathrm{p}}$ ” means using CircumIoU loss in Sec. 3.4, and otherwise, IoU loss [60] is used following a conversion from RBox to HBox (see H2RBox [20]).
Weights between  $L_{\mathrm{flp}}$  and  $L_{\mathrm{rot}}$ : Table 6 shows that on both DOTA and HRSC datasets,  $\lambda = 0.05$  could be the best choice under  $\mathrm{AP}_{50}$  metric, whereas  $\lambda = 0.1$  under  $\mathrm{AP}_{75}$ . Hence in most experiments, we choose  $\lambda = 0.05$ , except for Table 7 where  $\lambda = 0.1$  is used.
Range of view generation: When the rotation angle  $\mathcal{R}$  is close to 0, the SS branch could fall into a sick state. This may explain the fluctuation of losses under the random rotation within  $-\pi \sim \pi$  leading to training instability. According to Table 7,  $\pi /4\sim 3\pi /4$  is more suitable.
Table 6: Ablation with different weights between flipping and rotating losses defined in Eq. 6.
Dataset	λ	AP	AP50	AP75	Dataset	λ	AP	AP50	AP75
DOTA	0	31.60	66.37	25.03	HRSC	0	0.06	0.32	0.00
0.01	40.43	72.26	38.55	0.01	55.78	89.20	61.72
0.05	40.69	72.31	39.49	0.05	58.03	89.66	64.80
0.1	40.48	71.46	39.84	0.1	58.22	89.45	64.99
0.5	39.94	72.26	38.16	0.5	53.85	88.90	61.47
1.0	38.50	70.91	36.02	1.0	1.57	6.97	0.38
Table 7: Ablation with different random ranges in the rotated view generation on HRSC.
Range	AP	AP50	AP75
-π ~ π*	56.57	89.47	63.14
π/4 ~ 3π/4	58.22	89.45	64.99
3π/8 ~ 5π/8	56.81	89.83	64.03
7π/16 ~ 9π/16	55.56	89.40	61.28
* Not stable, occasionally be much lower.
Table 8: Ablation with different padding strategies for rotated view generation.
Dataset	Padding	AP	AP50	AP75
DOTA	Zeros	40.49	72.26	39.15
Reflection	40.69	72.31	39.49
HRSC	Zeros	55.90	89.32	60.95
Reflection	58.03	89.66	64.80
Table 9: Ablation with different levels of noise adding to HBox annotations on DOTA.
σ	H2RBox	H2RBox-v2
AP50	AP75	AP50	AP75
0%	70.05	38.38	72.31	39.49
10%	69.19	35.24	71.68	36.33
30%	67.39	26.02	71.11	34.12
Table 10: Ablation training with different sampling percentages of DOTA dataset.
p	H2RBox	H2RBox-v2
AP50	AP75	AP50	AP75
100%	70.05	38.38	72.31	39.49
30%	55.73	20.14	61.25	27.91
10%	37.71	6.98	44.61	14.97
Branch multiplexing: An additional experiment that randomly selects from  $5\%$  flip or  $95\%$  rotation in only one branch (the proportion based on  $\lambda = 0.05$  in Table 6) shows  $\mathrm{AP}_{50} / \mathrm{AP}_{75}$ :  $72.24\% /39.51\%$  (DOTA w/o MS) while reducing the training time and the RAM usage to H2RBox's level.
Padding strategies: Compared to the performance loss of more than  $10\%$  for H2RBox without reflection padding, Table 8 shows that H2RBox-v2 is less sensitive to black borders.
**Annotation noise:** Table 9 multiplies the height and width of annotated HBox by a noise from the uniform distribution  $(1 - \sigma, 1 + \sigma)$ . When  $\sigma = 30\%$ , the  $\mathrm{AP}_{50}$  of H2RBox-v2 drops by only  $1.2\%$ , less than H2RBox  $(2.69\%)$ , which demonstrates the better robustness of our method.
Training data volume: Table 10 displays that the gap between H2RBox and H2RBox-v2 becomes larger on the sampled version of DOTA, where  $p$  denotes the sampling percentage.
This paper presents H2RBox-v2, a weakly-supervised detector that learns the RBox from the HBox annotation. Unlike the previous version H2RBox, H2RBox-v2 learns the angle directly from the image of the objects through a powerful symmetry-aware self-supervised branch, which further bridges the gap between HBox-supervised and RBox-supervised oriented object detection.
Extensive experiments are then carried out with the following observations: 1) Compared to H2RBox, H2RBox-v2 achieves higher accuracy on various datasets, with an improvement of  $2.32\%$  on average over three versions of DOTA, and  $6.33\%$  on the FAIR1M dataset. 2) H2RBox-v2 is less susceptible to low annotation quality and insufficient training data. As a result, it is compatible with small datasets such as HRSC, which H2RBox cannot handle. 3) Even compared to fully-supervised counterpart (i.e. Rotated FCOS), H2RBox-v2 still shows quite competitive performance, proving the effectiveness and the potential of symmetry-aware self-supervision for rotating detection.
Broader impacts. Oriented object detection can be used for military purposes e.g. by remote sensing.
Limitations. There can be cases when the objects are not symmetric in appearance. This may also hold when the objects are partially occluded even from the top view. Moreover, it becomes more challenging to explore the symmetry in 3-D object rotation detection due to occlusion, which yet has been well explored in [39] for autonomous driving by the H2RBox (v1) method [20].
[57] Yanxi Liu. Computational symmetry. In Computer Vision: A Reference Guide. Springer International Publishing, 2020.
[58] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[59] Tsung-Yi Lin, Piotr Dólar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In IEEE Conference on Computer Vision and Pattern Recognition, pages 936-944, 2017.
[60] Jiahui Yu, Yuning Jiang, Zhangyang Wang, Zhimin Cao, and Thomas Huang. Unitbox: An advanced object detection network. In Proceedings of the 24th ACM International Conference on Multimedia, pages 516-520, 2016.
[61] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, pages 8024-8035, 2019.
[62] Yue Zhou, Xue Yang, Gefan Zhang, Jiabao Wang, Yanyi Liu, Liping Hou, Xue Jiang, Xingzhao Liu, Junchi Yan, Chengqi Lyu, Wenwei Zhang, and Kai Chen. MMrotate: A rotated object detection benchmark using pytorch. In Proceedings of the 30th ACM International Conference on Multimedia, 2022.
[63] Liping Hou, Ke Lu, Jian Xue, and Yuqiu Li. Shape-adaptive selection and measurement for oriented object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 923-932, 2022.
[64] Zonghao Guo, Chang Liu, Xiaosong Zhang, Jianbin Jiao, Xiangyang Ji, and Qixiang Ye. Beyond bounding-box: Convex-hull feature adaptation for oriented and densely packed object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8788-8797, 2021.
[65] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF International Conference on Computer Vision, pages 9992-10002, 2021.
[66] Xian Sun, Peijin Wang, Zhiyuan Yan, Feng Xu, Ruiping Wang, Wenhui Diao, Jin Chen, Jihao Li, Yingchao Feng, Tao Xu, Martin Weinmann, Stefan Hinz, Cheng Wang, and Kun Fu. Fair1m: A benchmark dataset for fine-grained object recognition in high-resolution remote sensing imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 184:116-130, 2022.
[67] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
[68] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.