SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yi Yu $^{1*}$ , Xue Yang $^{2,3*}$ , Qingyun Li $^{4}$ , Yue Zhou $^{2}$ , Gefan Zhang $^{5}$ , Feipeng Da $^{1\dagger}$ , Junchi Yan $^{2,3\dagger}$
$^{1}$ Southeast University  $^{2}$ MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University
$^{3}$ Shanghai AI Laboratory  $^{4}$ Harbin Institute of Technology  $^{5}$ COWAROBOT Co. Ltd.
{yuyi,dafp}@seu.edu.cn，21b905003@stu.hit.edu.cn
{yangxue-2019-sjtu,sjtu_zy,lizaozhouke,yanjunchi}@sjtu.edu.cn
PyTorch Code: https://github.com/open-mmlab/mmrotate
With the rapidly increasing demand for oriented object detection, e.g. in autonomous driving and remote sensing, the recently proposed paradigm involving weakly-supervised detector H2RBox for learning rotated box (RBox) from the more readily-available horizontal box (HBox) has shown promise. This paper presents H2RBox-v2, to further bridge the gap between HBox-supervised and RBox-supervised oriented object detection. Specifically, we propose to leverage the reflection symmetry via flip and rotate consistencies, using a weakly-supervised network branch similar to H2RBox, together with a novel self-supervised branch that learns orientations from the symmetry inherent in visual objects. The detector is further stabilized and enhanced by practical techniques to cope with peripheral issues e.g. angular periodicity. To our best knowledge, H2RBox-v2 is the first symmetry-aware self-supervised paradigm for oriented object detection. In particular, our method shows less susceptibility to low-quality annotation and insufficient training data compared to H2RBox. Specifically, H2RBox-v2 achieves very close performance to a rotation annotation trained counterpart - Rotated FCOS: 1) DOTA-v1.0/1.5/2.0:  $72.31\% /64.76\% /50.33\%$  vs.  $72.44\% /64.53\% /51.77\%$  ; 2) HRSC:  $89.66\%$  vs.  $88.99\%$  ; 3) FAIR1M:  $42.27\%$  vs.  $41.25\%$ .
Object detection has been studied extensively, with early research focusing mainly on horizontal detection [1, 2]. When fine-grained bounding boxes are required, oriented object detection [3] is considered more preferable, especially in complex scenes such as aerial images [4-8], scene text [9-13], retail scenes [14], and industrial inspection [15, 16].
With oriented object detection being featured, some horizontal-labeled datasets have been re-annotated, such as DIOR [17] to DIOR-R [18] for aerial image (192K instances) and SKU110K [19] to SKU110K-R [14] for retail scene (1,733K instances). Although re-annotation enables the training of oriented detectors, two facts cannot be ignored: 1) Horizontal boxes (HBoxes) are more readily available in existing datasets; 2) Rotated box (RBox) or Mask annotation are more expensive.
†Corresponding author. The work was partly supported by National Natural Science Foundation of China (62306069, 62222607), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), Special Project on Basic Research of Frontier Leading Technology of Jiangsu Province of China (BK20192004C), and China Postdoctoral Science Foundation (2023M740602). Yi Yu is also supported by Jiangsu Funding Program for Excellent Postdoctoral Talent (2023ZB616).
Figure 1: Visual comparisons of HBox-supervised oriented detectors, including BoxInst-RBox (2021) [21], BoxLevelSet-RBox (2022) [22], H2RBox (2023) [20] in the first row, SAM-RBox (2023) [23] in the second row, and H2RBox-v2 (our work) in the third row.
Such a situation raises an interesting question: Can we achieve oriented object detection directly under the weak supervision of HBox annotation? Yang et al. [20] explored this new task setting and proposed the first general solution called HBox-to-RBox (H2RBox) in 2023.
H2RBox gives an effective paradigm and outperforms potential alternatives, including HBox-Mask-RBox (generating RBoxes from segmentation mask) powered by BoxInst [21] and BoxLevelSet [22], the state-of-the-art HBox-supervised instance segmentation methods. Yet it is not impeccable at least in two folds: 1) H2RBox learns the angle from the geometry of circumscribed boxes, requiring high annotation quality and a large number of training data containing the same object in various orientations. 2) H2RBox requires quite restrictive working conditions. For example, it is incompatible with rotation augmentation and is sensitive to black borders on images.
If the above requirements are not met, H2RBox may perform far below expectations. As a result, H2RBox does not support some small datasets (e.g. HRSC [5]) and some augmentation methods (e.g. random rotation), leading to bottlenecks in its application scenarios and performance.
By rethinking HBox-supervised oriented object detection with a powerful yet unexplored theory, symmetry-aware self-supervision, we present H2RBox-v2, a new version of H2RBox that exploits the symmetry of objects and solves the above issues.
Motivation: Symmetry is a natural and popular property in vision [24]. For example, in the DOTA dataset [4], many categories (planes, courts, vehicles, ships, etc.) show significant reflection symmetry. For RBox annotation, symmetry is also an important consideration—Humans can intuitively take the direction of the symmetrical axis as the orientation, even without annotation guidance. Is there a way to use symmetry as a supervising tool in deep neural networks for oriented object detection? Will this technique lead to better performance? These are the questions for which this paper is intended.
What is new in H2RBox-v2? 1) A new self-supervised branch leaning angle from symmetry. H2RBox-v2 directly learns angles from the image through the symmetry of objects. That means it is capable of learning the angles correctly even if the HBox annotations are inaccurate in size (not
Table 1: Comparison between H2RBox [20] (denoted as "v1") and H2RBox-v2 (ours).
Arch.1	SS View Transform	Angle Acquisition Principle	Aug. Applicability	Dataset Applicability2	Gap3
MS	RR	DOTA	HRSC	FAIR1M
v1	WS+SS	Rotate	Geometric Constraints	✓	×	✓	×	✓	-3.41%
Rotate Consistency
v2	WS+SS	Flip	Symmetry-aware	✓	✓	✓	✓	✓	+0.07%
Rotate	Learning
<sup>1</sup> WS: Weakly-supervised, SS: Self-supervised. <sup>2</sup> Based on whether the training converges on the dataset.
3 Performance gap of  $\mathrm{AP}_{50}$  compared to RBox-supervision (i.e. FCOS) average on datasets with "√".
precisely the circumscribed rectangle, see Table 9) or when the training data is relatively insufficient (see Table 10). 2) A newly designed CircumIoU loss in the weakly-supervised branch. With this amendment, H2RBox-v2 is now compatible with random rotation augmentation. 3) As a result, H2RBox-v2 gives a higher performance (as is displayed in Fig. 1 and Table 1), further bridging the gap between HBox-supervised and RBox-supervised oriented object detection.
Contributions: 1) This work is the first attempt to explore symmetry for angle regression in oriented object detection, showing that symmetry in images can help learn the orientation of objects in a self-supervised manner. 2) A new training paradigm incorporating symmetry in images is elaborated on, and an integral and stable implementation is provided in this paper. 3) The proposed methods are evaluated via extensive experiments, showing their capability of learning the orientation via symmetry, achieving a higher accuracy than H2RBox. The source code is publicly available.
RBox-supervised oriented object detection: Representative works include anchor-based detector Rotated RetinaNet [25], anchor-free detector Rotated FCOS [26], and two-stage detectors such as RoI Transformer [27], Oriented R-CNN [28] and ReDet [29]. Besides,  $\mathbb{R}^3\mathrm{Det}$  [30] and  $\mathrm{S^2A - Net}$  [31] improve the performance by exploiting alignment features. Most of the above methods directly perform angle regression, which may face loss discontinuity and regression inconsistency induced by the periodicity of the angle. Remedies have been thus developed, including modulated losses [32, 33] that alleviate loss jumps, angle coders [34-36] that convert the angle into boundary-free coded data, and Gaussian-based losses [37-40] that transform rotated bounding boxes into Gaussian distributions. Additionally, RepPoint-based methods [41-43] provide new alternatives for oriented object detection, which predict a set of sample points that bounds the spatial extent of an object.
HBox-supervised instance segmentation: Compared with HBox-supervised oriented object detection, HBox-supervised instance segmentation, a similar task also belonging to weakly-supervised learning, has been better studied in the literature. For instance, SDI [44] refines the segmentation through an iterative training process; BBTP [45] formulates the HBox-supervised instance segmentation into a multiple-instance learning problem based on Mask R-CNN [46]; BoxInst [21] uses the color-pairwise affinity with box constraint under an efficient RoI-free CondInst [47]; BoxLevelSet [22] introduces an energy function to predict the instance-aware mask as the level set; SAM (Segment Anything Model) [23] produces object masks from input prompts such as points or HBoxes.
Most importantly, these HBox-supervised instance segmentation methods are potentially applicable to HBox-supervised oriented object detection by finding the minimum circumscribed rectangle of the segmentation mask. Such an HBox-Mask-RBox paradigm is a potential alternative for the task we are aiming at and is thus added to our experiments for comparison.
HBox-supervised oriented object detection: Though the oriented bounding box can be obtained from the segmentation mask, such an HBox-Mask-RBox pipeline can be less cost-effective. The seminal work H2RBox [20] circumvents the segmentation step and achieves RBox detection directly from HBox annotation. With HBox annotations for the same object in various orientations, the geometric constraint limits the object to a few candidate angles. Supplemented with a self-supervised branch eliminating the undesired results, an HBox-to-RBox paradigm is established.
Some similar studies use additional annotated data for training, which are also attractive but less general than H2RBox: 1) OAOD [48] is proposed for weakly-supervised oriented object detection. But in fact, it uses HBox along with an object angle as annotation, which is just "slightly weaker" than RBox supervision. Such an annotation manner is not common, and OAOD is only verified
Figure 2: The overview of H2RBox-v2, consisting of a self-supervised branch that learns angles from the symmetry of objects, and a weakly-supervised branch that learns other properties from HBoxes.
on their self-collected ITU Firearm dataset. 2) Sun et al. [49] proposes a two-stage framework: i) training detector with the annotated horizontal and vertical objects. ii) mining the rotation objects by rotating the training image to align the oriented objects as horizontally or vertically as possible. 3) KCR [50] combines RBox-annotated source datasets with HBox-annotated target datasets, and achieves HBox-supervised oriented detection on the target datasets via transfer learning.
Symmetry detection: Detecting the symmetry (e.g. reflection) has been a long-standing research topic in vision [24], with approaches including key point matching [51, 52], iterative optimization [53, 54], and deep learning [55, 56]. These methods are aimed at tasks quite different from this paper, and we are the first to introduce symmetry-aware learning for oriented object detection.
An overview of the proposed H2RBox-v2 is given in Fig. 2, which consists of a self-supervised (SS) branch (Sec. 3.2) and a weakly-supervised (WS) branch (Sec. 3.3).
3.1 Preliminaries and Approach Overview
In H2RBox-v2, SS branch is designed to learn the orientation of objects from symmetry via consistencies between several views of training images, whilst WS branch learns other properties from the HBox annotation. The core idea of symmetry-aware SS learning is described below.
Definition of reflection symmetry: An object has reflectional symmetry if there is a line going through it which divides it into two pieces that are mirror images of each other [57].
Assume there is a neural network  $f_{\mathrm{nn}}(\cdot)$  that maps a symmetric image  $I$  to a real number  $\theta$ :
$$
\theta = f _ {\mathrm {n n}} (I) \tag {1}
$$
To exploit reflection symmetry, we endow the function with two new properties: flip consistency and rotate consistency.
Property I: Flip consistency. With an input image vertically flipped,  $f_{\mathrm{nn}}(\cdot)$  gives an opposite output:
$$
f _ {\mathrm {n n}} (I) + f _ {\mathrm {n n}} (\operatorname {f l p} (I)) = 0 \tag {2}
$$
Figure 3: To illustrate that the reflection symmetry (Way 1) is equivalent to a vertical flip plus a rotation (Way 2). The expected outputs at the bottom show that  $f_{\mathrm{nn}}(I') = f_{\mathrm{nn}}(I_2) \Rightarrow \theta_{\mathrm{pred}} = \theta_{\mathrm{sym}}$ .
where  $\mathrm{flp}(I)$  is an operator of vertically flipping the image  $I$ .
Property II: Rotate consistency. With an input rotated by  $\mathcal{R}$ , the output of  $f_{\mathrm{nn}}(\cdot)$  also rotates by  $\mathcal{R}$ :
$$
f _ {\mathrm {n n}} \left(\operatorname {r o t} (I, \mathcal {R})\right) - f _ {\mathrm {n n}} (I) = \mathcal {R} \tag {3}
$$
where  $\operatorname{rot}(I, \mathcal{R})$  is an operator that clockwise rotates the image  $I$  by  $\mathcal{R}$ .
Now we consider that given an image  $I_0$  symmetric about  $\varphi = \theta_{\mathrm{sym}}$ , assuming the corresponding output is  $\theta_{\mathrm{pred}} = f_{\mathrm{nn}}(I_0)$ , the image can be transformed in two ways as shown in Fig. 3:
- Way 1: Flipping  $I_0$  along line  $\varphi = \theta_{\mathrm{sym}}$ . According to the above definition of reflection symmetry, the output remains the same, i.e.  $f_{\mathrm{nn}}(I^t) = f_{\mathrm{nn}}(I_0) = \theta_{\mathrm{pred}}$ .
- Way 2: Flipping  $I_0$  vertically to obtain  $I_1$  first, and then rotating  $I_1$  by  $2\theta_{\mathrm{sym}}$  to obtain  $I_2$ . According to flip and rotate consistencies, the output is supposed to be  $f_{\mathrm{nn}}(I_2) = -\theta_{\mathrm{pred}} + 2\theta_{\mathrm{sym}}$ .
On the ground that ways 1 and 2 are equivalent, the transformed images  $I'$  and  $I_2$  are identical. And thus  $f_{\mathrm{nn}}(I') = f_{\mathrm{nn}}(I_2)$ , finally leading to  $\theta_{\mathrm{pred}} = \theta_{\mathrm{sym}}$ .
From the above analysis, it arrives at a conclusion that if image  $I_0$  is symmetric about line  $\varphi = \theta_{\mathrm{sym}}$  and function  $f_{\mathrm{nn}}(\cdot)$  subjects to both flip and rotate consistencies, then  $f_{\mathrm{nn}}(I_0)$  must be equal to  $\theta_{\mathrm{sym}}$ , so the orientation is obtained. In the following, we further devise two improvements.
Handling the angle periodicity: For higher conciseness, the periodicity is not included in the above formula. To allow images to be rotated into another cycle, the consistencies are modified as:
$$
f _ {\mathrm {n n}} (I) + f _ {\mathrm {n n}} (\operatorname {f l p} (I)) = k \pi
$$
$$
f _ {\mathrm {n n}} (\operatorname {r o t} (I, \mathcal {R})) - f _ {\mathrm {n n}} (I) = \mathcal {R} + k \pi \tag {4}
$$
where  $k$  is an integer to keep left and right in the same cycle. This problem is coped with using the snap loss in Sec. 3.4. Meanwhile, the conclusion should be amended as:  $f_{\mathrm{nn}}(I_0) = \theta_{\mathrm{sym}} + k\pi /2$ , meaning that the network outputs either the axis of symmetry or a perpendicular one.
Extending to the general case of multiple objects: Strictly speaking, our above discussion is limited to the setting when image  $I_0$  contains only one symmetric object. In detail implementation, we use an assigner to match the center of objects in different views, and the consistency losses are calculated between these matched center points. We empirically show on several representative datasets (see Sec. 4.2) that our above design is applicable for multiple object detection with objects not perfectly symmetric, where an approximate axis of each object can be found via learning.
3.2 Self-supervised (SS) Branch
The above analysis suggests that the network can learn the angle of objects from symmetry through the flip and rotate consistencies. A self-supervised branch is accordingly designed.
During the training process, we perform vertical flip and random rotation to generate two transformed views,  $I_{\mathrm{flp}}$  and  $I_{\mathrm{rot}}$ , of the input image  $I$ , as shown in Fig. 2 (a). The blank border area induced by rotation is filled with reflection padding. Afterward, the three views are fed into three parameters: shared branches of the network, where ResNet50 [58] and FPN [59] are used as the backbone and the neck, respectively. The random rotation is in range  $\pi /4 \sim 3\pi /4$  (according to Table 7).
Similar to H2RBox, a label assigner is required in the SS branch to match the objects in different views. In H2RBox-v2, we calculate the average angle features on all sample points for each object and eliminate those objects without correspondence (some objects may be lost during rotation).
Following the assigner, an angle coder PSC [36] is further adopted to cope with the boundary problem. Table 4 empirically shows that the impact of boundary problem in our self-supervised setting could
(a) Snap loss: Snapping predictions to the nearest cycle of targets
(b) CircumIoU loss: Between RBox predictions and circumscribed RBox GTs
Figure 4: Illustration of the snap loss (for SS branch) and the CircumIoU loss (for WS branch).
be much greater than that in the supervised case, especially in terms of stability. The decoded angles of the original, flipped, and rotated views are denoted as  $\theta$ ,  $\theta_{\mathrm{flp}}$ , and  $\theta_{\mathrm{rot}}$ .
Finally, with formula in Sec. 3.4, the loss  $L_{\mathrm{flp}}$  can be calculated between  $\theta$  and  $\theta_{\mathrm{fp}}$ , whereas  $L_{\mathrm{rot}}$  between  $\theta$  and  $\theta_{\mathrm{rot}}$ . By minimizing  $L_{\mathrm{flp}}$  and  $L_{\mathrm{rot}}$ , the network learns to conform with flip and rotate consistencies and gains the ability of angle prediction through self-supervision.
3.3 Weakly-supervised (WS) Branch
The SS branch above provides the angle of objects. To predict other properties of the bounding box (position, size, category, etc.), a weakly-supervised branch using HBox supervision is further introduced. The WS branch in H2RBox-v2 is inherited from H2RBox, but has some differences:
1) In H2RBox, the angle is primarily learned by the WS branch, with an SS branch eliminating the undesired results. Comparably, H2RBox-v2 has a powerful SS branch driven by symmetry that can learn the angle independently, and thus the angle subnet is deleted from the WS branch.
2) H2RBox converts  $B_{\mathrm{pred}}$  to an HBox to calculate the IoU loss [60] with the HBox  $B_{\mathrm{gt}}$ , which cannot work with random rotation where  $B_{\mathrm{gt}}$  becomes an RBox. To solve this problem, CircumIoU loss (detailed in Sec. 3.4) is proposed in H2RBox-v2 to directly calculate  $L_{\mathrm{box}}$  between  $B_{\mathrm{pred}}$  and  $B_{\mathrm{gt}}$ , so that  $B_{\mathrm{gt}}$  is allowed to be an RBox circumscribed to  $B_{\mathrm{pred}}$ , as is shown in Fig. 2 (c).
Loss for SS branch: As described in Eq. (4), the consistencies encounter the periodicity problem in rotation. To cope with this problem, the snap loss  $\ell_s$  is proposed in this paper:
$$
\ell_ {s} \left(\theta_ {\text {p r e d}}, \theta_ {\text {t a r g e t}}\right) = \min  _ {k \in Z} \left(\operatorname {s m o o t h} _ {L 1} \left(\theta_ {\text {p r e d}}, k \pi + \theta_ {\text {t a r g e t}}\right)\right) \tag {5}
$$
where an illustration is displayed in Fig. 4 (a). The snap loss limits the difference between predicted and target angles to  $\pi /2$  so that the loss calculation upon the two decoded angles will not induce boundary discontinuity. The snap loss is used in both  $L_{\mathrm{flp}}$  and  $L_{\mathrm{rot}}$  as:
$$
L _ {\mathrm {f l p}} = \ell_ {\mathrm {s}} (\theta_ {\mathrm {f l p}} + \theta , 0)
$$
$$
\begin{array}{l} L _ {\text {r o t}} = \ell_ {\mathrm {s}} (\theta_ {\text {r o t}} - \theta , \mathcal {R}) \end{array}
$$
where  $L_{\mathrm{flp}}$  is the loss for flip consistency, and  $L_{\mathrm{rot}}$  for rotate consistency.  $\theta$ ,  $\theta_{\mathrm{flp}}$ , and  $\theta_{\mathrm{rot}}$  are the outputs of the three views described in Sec. 3.2.  $\mathcal{R}$  is the angle in generating rotated view.
With the above definitions, the loss of the SS branch can be expressed as:
$$
L _ {\mathrm {s s}} = \lambda L _ {\mathrm {f l p}} + L _ {\mathrm {r o t}} \tag {7}
$$
where  $\lambda$  is the weight set to 0.05 according to the ablation study in Table 6.
Loss for the WS branch: The losses in the WS branch are mainly defined by the backbone FCOS detector, including  $L_{\mathrm{cls}}$  for classification and  $L_{\mathrm{cn}}$  for center-ness. Different from RBox-supervised
3There are a series of targets with interval  $\pi$ , just like a series of evenly spaced grids. The snap loss moves prediction toward the closest target, thus deriving its name from "snap to grid" function.
methods,  $B_{\mathrm{gt}}$  in our task is an RBox circumscribed to  $B_{\mathrm{pred}}$ , so we re-define the loss for box regression  $L_{\mathrm{box}}$  with CircumIoU loss as:
$$
\ell_ {p} \left(B _ {\text {p r e d}}, B _ {\mathrm {g t}}\right) = - \ln \frac {B _ {\text {p r o j}} \cap B _ {\mathrm {g t}}}{B _ {\text {p r o j}} \cup B _ {\mathrm {g t}}} \tag {8}
$$
where  $B_{\mathrm{proj}}$  is the dashed box in Fig. 4 (b), obtained by projecting the predicted box  $B_{\mathrm{pred}}$  to the direction of ground-truth box  $B_{\mathrm{gt}}$ .
The CircumIoU loss enables H2RBox-v2 to use random rotation augmentation to further improve the performance (see Table 2), which is not supported by H2RBox.
Finally, the loss of the WS branch can be expressed as:
$$
L _ {\mathrm {w s}} = L _ {\mathrm {c l s}} + \mu_ {\mathrm {c n}} L _ {\mathrm {c n}} + \mu_ {\mathrm {b o x}} L _ {\mathrm {b o x}} \tag {9}
$$
where the hyper-parameters are set to  $\mu_{\mathrm{cn}} = 1$  and  $\mu_{\mathrm{box}} = 1$  by default.
Overall loss: The overall loss of the proposed network is the sum of the WS loss and the SS loss:
$$
L _ {\text {t o t a l}} = L _ {\mathrm {w s}} + \mu_ {\mathrm {s s}} L _ {\mathrm {s s}} \tag {10}
$$
where  $\mu_{\mathrm{ss}}$  is the weight of SS branch set to 1 by default.
3.5 Inference Phase
Although using a multi-branch paradigm in training (as shown in Fig. 2), H2RBox-v2 does not require the multi-branch or the view generation during inference.
Due to the parameter sharing, the inference only involves the forward propagation of backbone, angle head (from SS), and other heads (i.e. regression, classification, and center-ness from WS). The only additional cost of H2RBox-v2 during inference is the PSC decoding (compared to H2RBox). Thus, FCOS/H2RBox/H2RBox-v2 have similar inference speed (see Table 2).
Using PyTorch 1.13.1 [61] and the rotation detection tool kits: MMRotate 1.0.0 [62], experiments are carried out. The performance comparisons are obtained by using the same platforms (i.e. PyTorch/MMRotate version) and hyper-parameters (learning rate, batch size, optimizer, etc.).
DOTA [4]: DOTA-v1.0 contains 2,806 aerial images—1,411 for training, 937 for validation, and 458 for testing, as annotated using 15 categories with 188,282 instances in total. DOTA-v1.5/2.0 are the extended version of v1.0. We follow the default preprocessing in MMRotate: The high-resolution images are split into  $1,024 \times 1,024$  patches with an overlap of 200 pixels for training, and the detection results of all patches are merged to evaluate the performance.
HRSC [5]: It contains ship instances both on the sea and inshore, with arbitrary orientations. The training, validation, and testing set includes 436, 181, and 444 images, respectively. With preprocessing by MMRotate, images are scaled to  $800 \times 800$  for training/testing.