H2RBox-v2: Incorporating Symmetry for Boosting Horizontal Box Supervised Oriented Object Detection
Authors: Yi Yu
Xue Yang
Qingyun Li
Yue Zhou
Feipeng Da
Junchi Yan
Venue: NeurIPS 2023 poster (2023)
Keywords: Oriented object detection
self-supervised learning
TL;DR: -

Abstract: With the rapidly increasing demand for oriented object detection, e.g. in autonomous driving and remote sensing, the recently proposed paradigm involving weakly-supervised detector H2RBox for learning rotated box (RBox) from the more readily-available horizontal box (HBox) has shown promise. This paper presents H2RBox-v2, to further bridge the gap between HBox-supervised and RBox-supervised oriented object detection. Specifically, we propose to leverage the reflection symmetry via flip and rotate consistencies, using a weakly-supervised network branch similar to H2RBox, together with a novel self-supervised branch that learns orientations from the symmetry inherent in visual objects. The detector is further stabilized and enhanced by practical techniques to cope with peripheral issues e.g. angular periodicity. To our best knowledge, H2RBox-v2 is the first symmetry-aware self-supervised paradigm for oriented object detection. In particular, our method shows less susceptibility to low-quality annotation and insufficient training data compared to H2RBox. Specifically, H2RBox-v2 achieves very close performance to a rotation annotation trained counterpart -- Rotated FCOS: 1) DOTA-v1.0/1.5/2.0: 72.31%/64.76%/50.33% vs. 72.44%/64.53%/51.77%; 2) HRSC: 89.66% vs. 88.99%; 3) FAIR1M: 42.27% vs. 41.25%.

H2RBox-v2: Incorporating Symmetry for Boosting Horizontal Box Supervised Oriented Object Detection
Yi Yu' Xue Yang2.3, Qingyun Li Yue Zhou', Gefan Zhang", Feipeng Da Junchi 2.n Southeast University Moo Key Lab of Artificial Intelligence, Shanghai Jiao Tong University Shhnnghai Al Laboratory "Harbin Institute of Technology 'COWAROBOT Co. Ltd.
Cyuyi dafp))su..ee edu. en, 21b905003dstu. hitttoottttttttt..o..oo..oo.o..t - sigtuusjjuuzyy lizaozhouke,, janjunchi edu. cn PyTorch Code: https: thub, com/ openomplab..o..eeeemmemmemmmmmmm www.b/mmaabaam...mmm.m mmrootate
Abstract
With the rapidly increasing demand for oriented object detection, e.g. in au- tonomous driving and remote sensing, the recently proposed paradigm involving weakly-supervised detector H2RBox for learning rotated box (RBox) from the more avvilaleev horizontal box (HBox) has shown promise. This paper presents H2RBox-v2, to further bridge the gap between HBox-supervised and -Box-supervised oriented object detection. Specifically, we propose to leverage the reflection symmetry via flip and rotate consistencies, using weakly-supervised network branch similar to H2RBox, together with a novel self-supervised branch that learns orientations from the symmetry inherent in visual objects. The detector is further stabilized and enhanced by practical techniques to cope with periph- eral issues e.g. angular periodicity. To our best knowledge, H2RBox-v2 is the first symmetry-aware self-supervised paradigm for oriented object detection. particular, method shows less susceptibility low-quality annotation and insuf- ficient training data compared to H2RBox. Specifically, H2RBox-v2 achieves very close performance to rotation annotation trained counterpart Rotated FCOS: 1) DOTA-vv10// 1/2.00 72.31% HRSC: 89.66% 88.99%; FAIRIM, M: 42.27%
vs. 72.44%%/64.55%/55..77.. 2) 41.25%.
Introduction
Object detection has been studied extensively, with early research focusing mainly horizontal detection [1, 2]. When fine-grained bounding boxes are required, oriented object detection [3] considered more preferable, especially in complex scenes such aerial images [4-8], scene text [9-13], retail scenes [14], and industrial inspection [15, 16].
With oriented object detection being featured, some horizontal-labeled datasets have been re- annotated, such as DIOR [17] to DIOR-R [18] for aerial image (192K instances) and SKU110K [19] to SKUI 10K-R [14] for retail scene (1,733K instances). Although re re-annotation enables the training of oriented detectors, facts cannot be ignored: 1) Horizontal boxes (HBoxes) are more readily available in existing datasets; 2) Rotated box (RBox) or Mask annotation are more expensive.
* Equul contribution.
Corresponniig author. The work was partly supported by National Natural Science Foundation China (62306069, 62222607), Shanghai Municipal Science and Technology Major Project (2021SH2 Special Project Basic Research of Frontier Leading Technology of Jiangsu Province of China (BK20192004C) and China Postdoctoral Science Foundation (2023M740602). Yi Yu also supported by Jiangsu Funding Program for Excellent Postdoctoral Talent (2023ZB616).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

BoxInsttRBBo
Boxl www.eeeeeeeeeeeeee Box
H2RBox
H2RBA Box
SAM-RBox
SAM-RBox
SAM-RBox
SAM-RBox
H2RBox-v2
H2RBox-v2
H2RBox-v2
H2RBox-v2
Figure I: Visual comparisons of HBox-supervised oriented detectors, including BoxInst-RBox (2021) [21], BoxLevveSSe-RRBox (2022) [22), H2RBox (2023) [20] in the first row, SAM-RBox (2023) [23] in the second row, and H2RBox-v2 (our work) in the third row.
Such a situation raises an interesting question: Can we achieve oriented object detection directly under the weak supervision of HBox annotation? Yang et al. [20] explored this new task setting and proposed the first general solution called HBox-to-RBox (H2RBox) in 2023.
H2RBox gives an effective paradigm and outperforms potential alternatives, including HBox-Mask RBox (generating RBoxes from segmentation mask) powered by BoxInst [21] and BexLevelSet [22], the state-of-the-art HBox-supervised instance segmentation methods. Yet it is not impeccable at least two folds: 1) H2RBox learns the angle from the geometry of circumscribed boxes, requiring high annotation quality and large number of training data containing the same object in various orientations. 2) H2RBox requires quite restrictive working conditions. For example, is incompatible with rotation augmentation and is sensitive to black borders images.
If the above requirements are not met, H2RBox may perform far below expectations. As a result, H2RBox does not support some small datasets (e.g. HRSC [5]) and some augmentation methods (e.g. random rotation), leading to bottlenecks in its application scenarios and performance.
By rethinking HBBoxsuperviied oriented object detection with a powerful yet unexplored theory, symmetry-aware self-supervision, the symmetry of objects and solves the above issues.
present H2RBoxxv2. new version of H2RBox that exploits
Symmetry is natural and popular property in vision [24]. For example, in the DOTA dataset [4], many categories (planes, courts, vehicles, ships, etc.) show significant reflection symmetry. Ciniideration-Humnae
Motivation:
can intuitively take the direction of the symmetrical axis as the orientation, even without annotation guidance. Is there a way to use symmetry
For RBox annotation, symmetry is also important
supervising tool in deep neural networks for oriented object detection? Will this technique lead to better performance? These are the questions for which this paper is intended.
What is new in H2RBox-v2? 1) A new self-supervised branch leaning angle from symmetry. H2RBox-v2 directly learns angles from the image through the symmetry objects. That means is capable of learning the angles correctly even if the HBox annotations are inaccurate in size (not

---

Table 1: Comparison between H2RBox [20] (denoted as "vI") and H2RBox-v2 (ours).
Arch.' !
SS View
Angle Acquisition
Aug. Applicability
Dataset Applicability"
Gap²
Transform
Principle
MS
RR
DOTA
HRSC
FAIRM
Geometric Constraints
v1
WS+SS
Rotate
Rotate Consistency
x
X
v
-3.41%
Flip
Symmetry-aware
v2
WS+SS
v
+0.07%
Rotate
Learriigg
WS: Weakly-supervised, SS: Self-supervised. Based on whether the training converges on the dataset. Performance gap of AP5o compared to RBox-supervision (i.e. FCOS) average on datasets with "v".
precisely the circumscribed rectangle, see Table 9) or when the training data is relatively insufficient (see Table 10). 2) A newly designed CircumloU loss the weakly amendment, H2RBox-v2 is now compatible with random rotation augmentation. 3) As a result, H2RBox-v2 gives a higher performance (as is displayed in Fig. 1 and Table 1), further bridging the gap between -Bux-supervised and RBox-supervised oriented object detection.
branch. With this
Contributions: 1) This work the first attempt to explore symmetry for angle regression in oriented object detection, showing that symmetry in images can help learn the orientation of objects in self-supervised manner. 2) A new training paradigm incorporating symmetry in images is elaborated on, and an integral and stable implementation is provided in this paper. 3) The proposed methods are evaluated via extensive experiments, showing their capability of learning the orientation via symmetry, achieving higher accuracy than H2RBox. The source code is publicly available.
2 Related Work
Representative works include anchor-based detector Rotated RetinaNet [25]. anchor-free detector Rotated FCOS [26], and two-stage detectors such as Rol Transformer [27]. Oriented R-CNN [28] and ReDet [29]. Besides, De [30] and S A-et [31] improve the performance by exploiting alignment features. Most of the above methods directly perform angle regression, which may face loss discontinuity and regression inconsistency induced by the periodicity of the angle. Remedies have been thus developed, including modulated losses [32, 33] that alleviate loss jumps, angle coders [34-36] that convert the angle into boundary-free -rree coded data, and Gaussian- based losses [37-40] that transform rotated bounding boxes into Gaussian distributions. A Additionally, RepPoint-basee methods [41-43] provide new alternatives for oriented object detection, which predict set of sample points that bounds the spatial extent of an object.
RBox-supervised oriented object detection:
Compared with HBox -BBx--uperriied oriented object detec- tion, HBox-supervised instance segmentation, similar task also belonging weakly-supervised learning, has been better studied in the literature. For instance, SDI [44] refines the segmentation through iterative training process; BBTP [45] formulates the HBox -Box-supeeeised instance segmenta- tion into a multiple-instance learning problem based on Mask R-CNN [46]; BoxInst [21] uses the color-pairwise affinity with box constraint under an efficient Rol-free CondInst [47]: BoxLevelSet [22] introduces an energy function to predict the instance-awarr mask as the level set; SAM (Segment Anything Model) [23] produces object masks from input prompts such as points or HBoxes.
HBox-supervised instance segmentation:
Most importantly, these supervuee instance segmentation methods are potentially applicable to HBox-supervised oriented object detection by finding the minimum circumscribed rectangle of the segmentation mask. Such an HIBox-Mask-RMl paradigm is a potential alternative for the task we aiming at and is thus added our experiments for comparison.
Though the oriented bounding box can be obtained from the segmentation mask, such an -iBox-MakkRBBoo pipeline be less cost-effective. The seminal work H2RBox [20] circumvents the segmentation step and achieves RBox detection directly from HBox annotation. With HBox annotations for the same object in various orientations, the geometric constraint limits the object to a few candidate angles. Supplemented with a self-supervised branch eliminating the undesired results, an HBox-to-RBox paradigm is established.
HBox-supervised oriented object detection:
Some similar studies use additional annotated data for training, which are also attractive but less general than H2RBox: 1) OAOD [48] is proposed for weakly-supervised oriented object detection. But in fact, it uses HBox along with an object angle as annotation, which just "slightly weaker" than RBox supervision. Such annotation manner is not common, and OAOD is only verified
3

---

Jn
Flipped view
Rotated view
R
Random rotation Reflection padding
(a) View generation
Backbone
Shared
Backbone Neck
Shared
Backbone
Neck
Neck
Hxx22256
Hx Wx256
Hx22256
Flip consistency
Rotate consistency
Angle
Angle
Angle
HxWx33
HxWx3
HxWxx3
AngleCoder
AngleCoder
AngleCoder
HxWx1
(Ong + +0,0)
HxWx1
fros=2l( 8,
HxWx1
Angles learned from symmetry
0
(b) Sclf-supervised (SS) branch
Regression
BoxCoder
CircumloU loss
Fip
mit
HxWx4
IxWx5
Classification
B Laxex (B ppedt BR))
B
Las
Backhone Nock
HxWxC
Hx Hxx5666
Center-ness
HBox supervision
RBox predictions
L
(Shared)
HxWx1
(c) Weakly-supervised (WS) branch
Figure 2: The overview of H2RBox-v2, consisting of a sulf--upe branch that learns angles from the symmetry of objects, and a weakly-supervie branch that learns other properties from HBoxes.
on their self-collected ITU Firearm dataset. 2) Sun et al. [49] proposes two-stage framework: training detector with the annotated horizontal and vertical objects. ii) mining the rotation objects by rotating the training image align the oriented objects horizontally vertically possible. 3) KCR [50] combines RBox-annotated source datasets with #-Bo--nnotater target datasets, and achieves supx-sipe oriented detection on the target datasets via transfer learning.
Symmetry detection: Detecting the symmetry (e-g. reflection) has been a long- standinn research topic in vision [24], with approaches including key point matching [51, 52], iterative optimization [53, 54], and deep learning [55, 56]. These methods are aimed at tasks quite different from this paper, and we are the first to introduce symmetry-awan learning for oriented object detection.
Proposed Method
An overview of the proposed H2RBox-v2 is given Fig. 2, which consists self-supervised (SS) branch (Sec. 3.2) and weakly-supervised (WS) branch (Sec. 3.3).
3.1 Preliminaries and Approach Overview
In H2RBox-v2, SS branch designed to learn the orientation of objects from symmetry via consis- tencies between several views of training images, whilst WS branch learns other properties from the HBox annotation. The core idea of symmetry-aware SS learning is described below.
Definition of reflection symmetry: An object has reflectional symmetry if there is line going through it which divides into two pieces that are mirror images of each other [57].
Assume there is neural network fmn (-) that maps symmetric image I to a real number 0:
= (T)
(1)
To exploit reflection symmetry, we endow the function with two new properties: flip consistency and rotate consistency.
Property Flip consistency. With an input image vertically flipped, fmn (-) gives an opposite output:
fun (I) + fm (flp (f))
(2)

---

Way
Way
Flipped along @ Bagi
Flipped vertically
Rotated by 20
pred
(mn(1)
Axis of symmetry: @ 4yn
Pred
San
P 20 sy
Figure 3: To illustrate that the reflection symmetry (Way 1) is equivalent vertical flip plus
rotation (Way 2). The expected outputs at the bottom show that fun (1") = fun (12) + Opred = 0 sym
where flp (4) is an operator of vertically flipping the image 1.
Property II: Rotate consistency. With an input rotated by R. the output of fmn (+) also rotates by R:
San (rot (I, R.) Sm (7) =
where rot (I, R) is operator that clockwise rotates the image by R.
Now we consider that given an image symmetric about yp = Osym: assuming the corresponding output is Pped Jm (T)), the image can be transformed in two ways shown in Fig. 3:
Flipping do along line &p- Osym- According to the above definition of reflection symmetry, the output remains the same, i.c. fon (1') fmn (lo) = pred-
Way 1:
Flipping Io vertically obtain I first, and then rotating Iı by 28.ym to obtain Iz. According to flip and rotate consistencies, the output is supposed to be fmn (I2) = 8pred + 20
Way 2:
On the ground that ways and 2 are equivalent, the transformed images I' and 12 are identical. And thus fmm (I') fm (T2), finally leading to 0pred Osym-
From the above analysis, it arrives at a conclusion that if image To is symmetric about line 4 = Oxym and function fm (.) subjects both flip and rotate consistencies, then Jun (To) must be equal Asym so the orientation is obtained. In the following, we further devise two improvements.
Handling the angle periodicity: For higher conciseness, the periodicity is not included in the above formula. To allow images to be rotated into another cycle, the consistencies are modified as:
Jm (T) fun (flp (T)) =kx
fm (rot (T) - Fm (T) -R + km
where & is an integer keep left and right in the same cycle. This problem coped with using the snap loss in Sec. 3.4. Meanwhile, the conclusion should be amended as: fmm (To) Osym + kv kr/22, meaning that the network outputs either the axis of symmetry or perpendicular one.
Extending to the general case of multiple objects: Strietly speaking, our above discussion is limited the setting when image lo contains only one symmetrie object. In detail implementation, we use an assigner to match the center of objects in different views, and the consistency losses are calculated between these matched center points. We empirically show on several representative datasets (see Sec. 4.2) that our above design is applicable for multiple object detection with objects not perfectly symmetric, where an approximate axis of each object can be found via learning.
3.2 Self-supervised (SS) Branch
The above analysis suggests that the network can learn the angle of objects from symmetry through the flip and rotate consistencies. A self-supervised branch is accordingly designed.
During the training process, we perform vertical flip and random rotation to generate two transformed views, Inp and Lor, of the input image L, as shown in Fig. 2 (a). The blank border area induced by rotation is filled with reflection padding. Afterward, the three views are fed into three parameter- shared branches of the network, where ResNet50 [58] and FPN [59] are used the backbone and the neck, respectively. The random rotation is in range /4 ~ 3T/ / (according to Table 7).
Similar to H2RBox, label assigner is required in the SS branch to match the objects in different views. In H2RBox-v2, we calculate the average angle features on all sample points for each object and eliminate those objects without correspondence (some objects may be lost during rotation).
Following the assigner, an angle coder PSC [36] further adopted to cope with the boundary problem. Table 4 empirically shows that the impact of boundary problem in our self-supervised setting could
5

---

Periodic targets
Projecting B prd to g is direction
(2k 1)x+ Ourge
pesi())
Lossig)
(0
prea
Predicted box
Opreee)
Bet
Ground-trutd tound box
Losse)
2k Ouage
Batoj Projected box
Ittp Lrat
Intersection area
poree uagge))
ppeat
min(smmm (smooth poeas kn 0 easget) Ke
L = Eg( prods BB)) In
(a) Snap loss:
Snapping predictions to the nearest eycle of targets
(b) CircumloU loss: Between RBox predictions and circumseribed RBox GTs
Figure 4: Illustration of the snap loss (for SS branch) and the CircumloU loss (for WS branch).
be much greater than that in the supervised case, especially in terms of stability. The decoded angles of the original, flipped, and rotated views denoted 0, Oap. and P
Finally, with formula in Sec. 3.4, the loss Lip can be calculated between and 0np. whereas Trot between 0 and f By minimizing Lip and Trxs the network learns to conform with flip and rotate consistencies and gains the ability angle prediction through self-supervision.
3.3 Weakly-supervised (WS) Branch
The SS branch above provides the angle of objects. To predict other properties of the bounding box (position, size, category, etc.), a weakly-supervised branch using HBox supervision is further introduced. The WS branch in H2RBox-v2 is inherited from H2RBox, but has some differences:
1) In H2RBox, the angle is primarily learned by the WS branch, with an SS branch eliminating the undesired results. Comparably, H2RBox-v2 has powerful SS branch driven by symmetry that can learn the angle independently. and thus the angle subnet is deleted from the WS branch.
2) H2RBox converts B Bpred to an HBox to calculate the IoU loss [60] with the HBox Bg. which cannot work with random rotation where Ber becomes an RBox. To solve this problem, CircumloU loss (detailed in Sec. 3.4) is proposed in H2RBox-v2 to directly calculate Loox between Bpred and gg, so that Bg is allowed to be an RBox circumscribed to Bpred, as shown Fig. (c).
3.4 Loss Functions
Loss for SS branch: As described in Eq. (4), the consistencies encounter the periodicity problem in rotation. To cope with this problem, the snap loss Eg is proposed in this paper:
Lg (ppred, uurget = min (smoothuı (ppred, kx Ourget)) KEZ
(5)
where illustration is displayed in Fig. 4 (a). The snap loss limits the difference between predicted and target angles to 7/2 so that the loss calculation upon the two decoded angles will not induce boundary discontinuity. The snap loss is used in both Lup and Lyot as:
Lap Eg (Omp + 0,0) 0) L = f, (O R)
(6)
where Tap is the loss for flip consistency, and Traa for rotate consistency. 0, Opps and 0no are the outputs of the three views described in Sec. 3.2. R is the angle in generating rotated view.
With the above definitions, the loss of the SS branch can be expressed as:
Lys
XL Lioi
where À is the weight set to 0.05 according to the ablation study in Table 6.
Loss for the WS branch: The losses in the WS branch mainly defined by the backbone FCOS detector, including Lus for classification and Len for center-ness. Different from RBox-supervised
'There are a series of targets with interval x, just like series of evenly spaced grids. The snap loss moves prediction toward the closest target, thus deriving its name from "snap grid" function.

---

methods, Bg in our task is an RBox circumscribed to Bpred, so we re-define the loss for box regression Loox with CircumloU loss as:
B prog n Bg Bproj Bgt
Ep (Beo Bgt) in
where Bproj the dashed box in Fig. 4 (b), obtained by projecting the predicted box Bpred to the direction of ground-truth box Bgr.
The CircumloU loss enables H2RBox-v2
random rotation augmentation to further improve the performance (see Table 2), which not supported by H2RBox.
Finally, the loss of the WS branch can be expressed as:
Lws Lets + Men + Phox Lhox
(9)
where the hyper-parameters are set to Mee and Whox 1 by default.
Overall loss: The overall loss of the proposed network is the of the WS loss and the SS loss:
Luotal Loys + Pas Lus
(10)
where yss is the weight of SS branch set to by default.
3.5 Inference Phase
Although using multi-branch paradigm in training (as shown in Fig. 2), H2RBox-v2 does not require the multi-branch or the view generation during inference.
Due to the parameter sharing, the inference only involves the forward propagation of backbone, angle head (from SS), and other heads (i.e. regression, classification, and center-ness from WS). The only additional cost of H2RBox-v2 during inference is the PSC decoding (compared to H2RBox). Thus, FCOS/H2RBox/H2RBOx have similar inference speed (see Table 2).
4 Experiments
Using PyTorch 1.13.1 [61] and the rotation detection tool kits: MMRotate 1.0.0 [62], experiments are carried out. The performance comparisons are obtained by using the same platforms (i.e. PyTorch/MMMRRtate version) and hyper- parameters (learnngg rate, batch size, optimizer, etc.).
4.1 Datasets and Settings
DOTA [4]: DOTA-v1.0 0 contains 2,806 aerial 1nages-11,111 for training, 937 for validation, and 458 for testing, annotated using categories with 188.282 instances in total. DOTA-v1.5// are the extended version of v1.0. We follow the default preprocessing in MMRotate: The high-resolution images are split into 1,024 1,024 patches with an overlap of 200 pixels for training, and the detection results of all patches are merged to evaluate the performance.
HRSC [5]: It contains ship instances both the sea and inshore, with arbitrary orientations. The training, validation, and testing set includes 436, 181, and 444 images, respectively. With preprocessing by MMRotate, images are scaled 800 800 for training/testing.
FAIRIM [66]: It contains more than million instances and more than 40,000 images for fine- grained object recognition in high-resolution remote sensing imagery. The dataset is annotated with five categories and 37 fine- tne-grained subcategories. We split the images into 1,024 x 1,024 patches with overlap of 200 pixels and scale rate of 1.5 and merge the results for testing. The performance is evaluated on the FAIRIM-LO. server.
Experimental settings: We adopt the FCOS [26] detector with ResNet50 [58] backbone and FPN [59] neck as the baseline method, based on which we develop our H2RBox-v2. We choose average precision (AP) as the primary metric to compare with existing literature. For fair comparison, all the listed models are configured based on ResNet50 [58] backbone and trained on NVIDIA RTX3090/4090 GPUs. All models are trained with AdamW [67], with an initial learning rate of 5e-5 and a mini-batch size of Besides, we adopt learning rate warm-up for 500 iterations, and the learning rate is divided by ten at each decay step. "Ix", "3x", and "6x" schedules indicate 12, 36, and 72 epochs for training. "MS" and "RR" denote multi-scale technique [62] and random rotation augmentation. Unless otherwise specified, "6x" is used for HRSC and "Ix" for the other datasets, while random flipping is the only augmentation that is always adopted by default.
7

---

Table 2: Results on the DOTA-v1.0 dataset.
Method
Sched.
MS
RR
Size
FPS
AP50
RepPoints (2019) [41]
1x
1,024
24.5
68.45
RetinaNet (2017) [25]
1,024
25.4
68.69
KLD (2021) [38]
Ix
1,024
71.24
KFloU (2023) [40]
Ix
1,024
25.4
71.61
RBox- RBox- superisee
GWD (2021) [37]
1x
1,024
25.4
71.66
PSC (2023) [36]
Ix
1,024
25.4
71.92
SASM (2022) [63]
1,024
24.4
72.30
D' (2021) [30]
Ix
1,024
20.0
73.12
CFA (2021) [64]
1x
1,024
24.5
73.84
Oriented RepPoints (2022) [43]
1,024
24.5
75.26
S AeA-Net (2022) [31]
1x
1,024
23.3
75.81
FCOS (2019) [26]
IX
1,024
29.5
72.44
FCOS (2019) [26]
3x
1,024
29.5
74.75
FCOS (2019) [26]
1,024
29.5
77.68
BoxInst-RBox (2021)
1x
960
2.7
53.59
BoxLevelSet-RBox (2022) [22]
960
4.7
56.44
SAM-ViT-B-RBox (2023) [23]
1x
1,024
1.7
63.94
H2RBox (FCOS-based) (2023) [20]
1x
1,024
29.1
67.82
HBBx superviseeed
H2RBox hasos- (2023) [20]
1x
1,024
29.1
70.05
H2RBox SECCOS-based (2023) [20]
1X
1,024
29.1
74.40
H2RBox (FCOS- based) (2023) [20]
1x
1,024
29.1
75.35
H2RBox-v2 (FCOS-based)
1x
960
31.6
71.46
H2RBox-v2 (FCOS-based)
1x
1,024
29.1
72.31
H2RBox-v2 (FCOS-based)
3x
1,024
29.1
74.29
H2RBox-v2 (FCOS-based)
1x
1,024
29.1
77.97
H2RBox-v2 (FCOS-based)
Ix
1,024
29.1
78.25
H2RBox-v2 (FCOS-based) Swin-T)
1x
1,024
24.0
79.39
H2RBox-v2 (FCOS-based) Swin-B)
1x
1,024
80.61
RBox" means the minimum rectangle operation is performed on the Mask to obtain RBox.
Evaluated on NVIDIA V100 GPU due the excessive RAM usage.
The code available at https::/ //gitub/ com/ (im/Li- 1ii--innyuu//m samm mmrotate.
Results reported in the original paper [20].
Results reproduced by us with same infrastructure fair comparison.
Using Swin Transformer [65] as backbone four NVIDIA A100 GPUs with batch size = 4.
4.2 Main Results
DOTA-v1.0: Table shows that H2RBox-v2 outperforms HBox-Mask-Rboox methods in both accuracy and speed. Taking BoxL_evelSet-RBon [22] as an example, H2RBox-v2 gives an accuracy of 15.02% higher, and x7 speed faster by avoiding the time-consuming circumscribed rectangle operation). In particular, the recent foundation model for segmentation i.c. SAM [23] has shown strong zero-shot capabilities by training on the largest segmentation dataset to date. Thus, we use trained horizontal FCOS detector to provide HBoxes into SAM as prompts, so that the corresponding masks be generated by zero-shot, and finally the rotated Roxxes obtained by performing the minimum circumscribed rectangle operation on the predicted Masks. Thanks to the powerful zero-shot capability, SAM- based VIT-B [68] in Table 2 has achieved 63.94%. However, it is also limited to the additional mask prediction step and the time-consuming post-processing. only 1.7 FPS during inference.
pesse-poocssing (i.e. minimum
In comparison with the current state-of-the- art method H2RBox, to make it fair, we use the reproduced result of H2RBox, which achieves 70.05%, %. 2.23% 3% higher than the original paper [20]. In this fair comparison, our method outperforms H2RBox by 2.26% (72.31% % vs. 70.05%, both w/o MS). When MS is applied, the improvement is 2.62% (75.35% vs. 77.97%, both w/ MS).
Furthermore, the performance gap between our method and the RBox-supervised FCOS baseline is only 0.13% (w/o MS and RR) and 0.46% (w/ RR). When MS and RR are both applied, our method outperforms RBox-supervised FCOS by 0.57% (78.25% vs. 77.68%) proving that supplemented with symmetry-aware learning, the weakly-supervise with the fully-supervised one upon the same backbone neural network. Finally, H2RBox-v2 obtains 80.61% % DOTA-v1.0 by further utilizing stronger backbone.
learning can achieve performance par

---

Table 3: AP performance on the DOTA-v1.0/1.522.00 HRSC, and FAIRIM IM datasets.
Method
DOTA-v1.0
DOTA-v1.
DOTA-v2.0
HRSC
FAIRIM
RetinaNct (2017)(25)
68.69
60.57
47.00
84.49
37.67
GWD (2021) [37]
71,66
63.27
48.87
86.67
39.11
A-A-Net (2022) [31]
75.81
66.53
52.39
90.10
42.44
FCOS (2019) [26]
72.44
64.53
51.77
88.99
41.25
Sun al. (2021) [49]'
38.60
KCR (2023) [50]
79.10
H2RBox (2023) [20]
70.05
61.70
48.68 50.33
7.03
35.94
Box-v2
72.31
64.76
89.66
42.27
Sparse annotation for horizontal/vertical objects. The result cited from their paper.
Transfer learning from DOTA (RBox) to HRSC (HBBox). The result is cited from their paper.
Table 4: Ablation with different SS losses.
Table Ablation with different WS losses.
Dataset
PSC
4
AP
APso
AP
Dataset f
RR
AP
AP50
AP
24.24
52.24
19.48
39.35
71.49
37.03
0.01
0.77
0.02
11.93
29.34
7.86
DOTA
DOTA
V
10.49
27.57
6.15
/
40.69
72.31
39.49
40.69
72.31
39.49
V
40.17
71.79
39.77
2,25
7.83
0.62
56,20
89.58
61.84
48.95
88.52
50.03
41.10
87.19
33.97
HRSC
HRSC
0.31
0.88
0.13
58.03
89.66
64.80
58.03
89.66
64.80
/
63.82
89.56
76.11
DOTA-v1.5/2.0: As extended versions of DOTA-v1.0, these two datasets are more challenging. while the results present similar trend. Still, H2RBox-v2 shows considerable advantages over H2RBox, with an improvement of 3.06% on DOTA-v1 1.5 and 1.65% on DOTA-v2.0. The results on DOTA-v1.5/2.0 HRSC, and FAIR 1M are shown in Table 3.
HRSC: H2RBox [20] can hardly learn angle information from small datasets like HRSC, resulting in deficient performance. Contrarily, H2RBox-v2 is good this kind of task, giving performance comparable to fully-supervised methods. Compared to KCR [50] that uses transfer learning from RBox-supervised DOTA to HBox-supervised HRSC, our method, merely using HBox-supervised HRSC, outperforms KCR by 10.56% (89.66% vs. 79.10%)
This dataset contains large number of planes, vehicles, and courts, which are more perfectly symmetric than objects like bridges and harbors DOTA. This may explain the observation that H2RBox-v2, learning from symmetry, outperforms H2RBox by more considerable margin of 6.33% (42.27% vs. 35.90 35.94%). this case, H2RBox-v2 even performs superior to the fully-supervised FCOS that H2RBox-v2 is based on by 1.02% (42.27% vs. 41.25%)
FAIR1M:
4.3 Ablation Studies
Loss in SS branch:
Table 4 studies the impact using the snap loss (see Sec. 3.4) and the angle coder. Column "PSC" indicates using PSC angle coder [36] and "w/o PSC" means the conv layer directly outputs the angle. Column "L," with check mark denotes using snap loss (otherwise using smooth L1 loss). Without these two modules handling boundary discontinuity, we empirically find that the loss could fluctuate wide range, even failure convergence (see much lower results in Table 4). In comparison, when both PSC and snap loss are used, the training is stable.
Loss in WS branch: Table shows that CircumloU loss with random rotation can further improve the performance, which H2RBox is incapable of. "4p means using CircumloU loss in Sec. 3.4, and otherwise, IoU loss [60] used following a conversion from RBox HBox (see H2RBox [20]).
Weights between Lip and L Table shows that on both DOTA and HRSC datasets, = 0.05 could be the best choice under AP metrie, whereas X 0.1 under AP7s. Hence in most experiments, we choose 0.05, except for Table where = 0.1 is used.
Range of view generation: When the rotation angle R. is close to 0, the SS branch could fall into a sick state. This may explain the fluctuation of losses under the random rotation within -x ~ n, leading to training instability. According to Table 7, TT/ 3T is more suitable.

---

Table 6: Ablation with different weights between flipping and rotating losses defined in Eq. 6.
Dataset
AP
AP
Dataset
AP
AP7s
0
31.60)
66.37
25.03
0.06
0.32
0.00
0.01
40.43
72.26
38.55
0.01
55.78
89.20
61.72
0.05
40.69
72.31
39.49
0.05
58.03
89.66
64.80
DOTA
HRSC
0.1
40.48
71.46
39.84
58.22
89.45
64.99
0.5
39.94
72.26
38.16
0.5
53.85
88,90
61.47
1.0
38.50
70.91
36.02
1.0
1.57
6.97
0.38
Table 7: Ablation with different random ranges
Table 8: Ablation with different padding strate- gies for rotated view generation.
in the rotated view generation on HRSC.
Range
AP
50
AP75
Dataset
Padding
A.so
APrs
-I
56.57
89.47
63.14
DOTA
Zeros
40.49
72.26
39.15
T/4 ~ 3T/4
58.22
89.45
64.99
Reflection
40.69
72.31
39.49
3n 58/8
56.81
89.83
64.03
Zeros
55.90
89.32
60.95
HRSC
7 76/16 91/16
55.56
89.40
61.28
Reflection
58.03
89.66
64.80
Not stable, occasionally much lower.
Table 9: Ablation with different levels of noise adding to HBox annotations on DOTA.
Table 10: Ablation training with different sam- pling percentages of DOTA dataset.
H2RBox
H2RBox-v2
H2RBox
H2RBox-v2
APsu
AP
APso
AP7o
APsa
AP7s
APso
APrs
0%
70.05
38.38
72.31
39.49
100%
70.05
38.38
72.31
39.49
10%
69.19
35.24
71.68
36.33
30%
55.73
20.14
61.25
27.91
30%
67.39
26.02
71.11
34.12
10%
37.71
6.98
44.61
14.97
Branch multiplexing: An additional experiment that randomly selects from 5% flip or 95% rotation in only one branch (the proportion based on À 0.05 in Table 6) shows APto/AP7s: 22.24% (DOTA w/o MS) while reducing the training time and the RAM usage to H2RBox's level.
Padding strategies: Compared to the performance loss of more than 10% for H2RBox without reflection padding, Table 8 shows that H2RBox-v2 less sensitive black borders.
Annotation noise: Table 9 multiplies the height and width of annotated HBox by a noise from the uniform distribution (1 a, 1 rr). When 30%, the AP50 of H2RBox-v2 drops by only 1.2%, less than H2RBox (2.69%), which demonstrates the better robustness of our method.
Training data volume: Table 10 displays that the gap between H2RBox and H2RBox-v2 becomes larger on the sampled version of DOTA, where p denotes the sampling percentage.
5 Conclusion
This paper presents H2RBox-v2, a weakly-supervised detector that learns the RBox from the HBox annotation. Unlike the previous version H2RBox, H2RBox-v2 learns the angle directly from the image of the objects through a powerful symmetry-aware self-supervised branch, which further bridges the gap between HBox-supervised and RBox-supervised oriented object detection.
Extensive experiments are then carried out with the following observations: 1) Compared to H2RBox, H2RBox-v2 achieves higher accuracy various datasets, with improvement of 2.32% average over three versions of DOTA, and 6.33% on the FAIRIM IM dataset. 2) H2RBox-VV v2 is less susceptible to low annotation quality and insufficient training data. As result, is compatible with small datasets such as HRSC, which H2RBox cannot handle. 3) Even compared to fully-supervised counterpart (i.e. Rotated FCOS), H2RBox-v2 still shows quite competitive performance, proving the effectiveness and the potential of symmetry-award self-supervision for rotating detection.
Broader impacts. Oriented object detection can be used for military purposes e.g. by remote sensing.
Limitations. There can be cases when the objects are not symmetric in appearance. This may also hold when the objects are partially occluded even from the top view. Moreover, becomes more challenging explore the symmetry in 3-D object rotation detection due to occlusion, which yet has been well explored in [39] for autonomous driving by the H2RBox (yl) method [20].
10

---

References
Zhong- Qiu Zhao, Peng Zheng. Shou-Tao Xu, and Xindong Wu. Object detection with deep learning: review. IEEE Transactions on Neural Networks and Learning Systems, 30(11):3212-33 2019.
[2] Li Liu, Wanli Ouyang, Xiaogang Wang. Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen, Deep learning generic object detection: survey. International Journal of Computer Vision, 128(2):261-18, 2020.
[3] Long Wen, Yu Cheng, Yi Fang, and Xinyu Li. A comprehensive survey of oriented object detection in remote sensing images. Expert Systems with Applications, page 1 119960, 2023.
[4] Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jieho Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in aerial images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3974-3983, 2018.
[5] Zikun Liu, Liu Yuan, Lubin Weng, and Yiping Yang. A high resolution optical satellite image dataset for ship recognition and some new baselines. Proceedings of the International Conference Pattern Recognition Applications and Methods, volume 2. pages 324 331,, 2017.
[6] Xue Yang, Hao Sun, Kun Fu, Jirui Yang, Xian Sun, Menglong Yan, and 7hi Guo. Automatic ship detection in remote sensing images from google carth of complex scenes based on multiscale rotation dense feature pyramid networks. Remote Sensing, 10(1). 2018.
[7] Xue Yang and Junchi Yan. On the ihoitrary-orienter object detection: Classification based approaches revisited, International Journal Computer Vision, 130:1340- 136001135,0000000 2022.
[8] Kun Fu, Zhonghan Chang. Yue Zhang, Guangluan Xu, Keshu Zhang, and Xian Sun. Rotation-aware and multi-scale convolutional neural network for object detection in remote sensing images. ISPRS Journal of Photogrammetry and Remote Sensing. 161:294-3000 2020.
[9] Nibal Nayef, Fei Yin, Imen Bizid, Hyunsoo Choi, Yuan Feng, Dimosthenis Karatzas, Zhenbo Luo, Umapada Pal, Christophe Rigaud, Joseph Chazalon, Wafa Khlif, Muhammad Muzzamil Luqman, Jean- Christophe Burie, Cheng-lin Liu, and Jean-Mare Ogier. Tedar2017 rohust reading challenge on multi-lingual scene text detection and script identification rrc-mlt. In IAPR International Conference on Document Analysis and Recognition, volume 01, pages 1454-1459, 2017.
[10] Jianqi Ma, Weiyuan Shao, Hao Ye, Li Wang. Hong Wang. Yingbin Zheng, and Xiangyang Xue, Arbitrary- oriented scene text detection via rotation proposals. IEEE Transactions on Multimedia, 20(1)):3111333 11122, 2018.
[11] Minghui Liao, Zhen Zhu, Baoguang Shi, Gui-Song Xia, and Xiang Bai. Rotation-sensitio regression for oriented scene text detection. In IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 5909-5918. 2018
[12] Xuebo Liu, Ding Liang, Shi Yan, Dagui Chen, Yu Qiao, and Junjie Yan. Fots: Fast oriented text spotting with unified network. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5676-5685, 2018.
[13] Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. East: An efficient and accurate scene text detector. In IEEE Conference on Computer Vision and Pattern Recognition. pages 2642-2651 2017.
[14] Xingjia Pan, Yuqiang Ren, Kekai Sheng, Weiming Dong, Haolei Yuan, Xiaowei Guo, Chongyang Ma, and Changsheng Xu. Dynamic refinement network for oriented and densely packed object detection. In IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 11 1000 11216, 2020.
[15] Yuekai Liu, Hongli Gao, Liang Guo, Aoping Qin, Canyu Cai, and Zhichao You. A data-flow oriented deep ensemble learning method for real- time surface defect inspection. IEEE Transactions Instrumentation and Measurement, 69(7):4681 4691, 2020.
[16] Hongjin Wu, Ruoshan Lei, and Yibing Peng. Pchnet: defect inspection surface mount technology. IEEE Transactions on Instrumentation and Measurement, 71:1-14, 2022.
A lightweight convolutional neural network for
[17] Ke Li. Gang Wan, Gong Cheng. Liqiu Meng, and Junwei Han. Object detection in optical remote sensing images: A survey and new benchmark. ISPRS Journal of Photogrammetry and Remote Sensing, 159:296 307, 2020.
11

---

[18] Gong Cheng, Jiabao Wang, Ke Li, Xingxing Xie, Chunbo Lang, Yanqing Yao, and Junwei Han. Anchor- free oriented proposal generator for object detection. IEEE Trunsactions on Geoscience and Remote Sensing, 2022.
[19] Eran Goldman, Roei Herzig, Aviv Fisenschtat, Jacob Goldherger, and Tal Hassner. Precise detection in densely packed scenes. In /EEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5227-5236, 2019.
[20] Xue Yang, Gefan Zhang, Wentong Li, Xuchui Wang, Yue Zhou, and Junchi Yan. H2rbox: Horizontal box annotation is all you need for oriented object detection. International Conference on Learning Representations, 2023.
[21] Zhi Tian, Chunhua Shen, Xinlong Wang, and Hao Chen, Boxinst: High- instance segmentation with box annotations. In IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 5443-5452, 2021.
[22] Wentong Li, Wenyu Liu, Jianke Zhu, Miaomiao Cui, Xiansheng Hua, and Lei Zhang. Box- sux-eupee instance segmentation with level set evolution. In European Conference on Computer Vision, 2022.
[23] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao. Spencer Whitehead, Alexander C. Berg. Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. urXiv:2304.0260 2023.
[24] Christopher Funk, Seungkyu Lee, Martin R. Oswald, Stavros Tsogkas, Wei Shen, Andrea Cohen, Sven Dickinson, and Yanxi Liu. 2017 7 iccv challenge: Detecting symmetry in the wild. In IEEE International Conference on Computer Vision Workshops, pages 1692-1701, 2017.
[25] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2):318-33 121111 2020.
[26] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In IEFF//VF International Conference on Computer Vision, pages 9626-9635, 2019.
[27] Jian Ding, Nan Xue, Yang Long, Gui-Song Xia, and Qikai Lu. Learning roi transformer for oriented object detection in aerial images. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2849 2858, 2019.
[28] Xingxing Xie, Gong Cheng, Jiabao Wang, Xiwen Yao, and Junwei Han. Oriented r-cnn for object detection. In IEEE/CVF International Conference on Computer Vision, pages 3520-3529, 2021.
[29] Jiaming Han, Jian Ding, Nan Xue, and Gui-Song Xia. Redet: A rotation equivariant detector for aerial object detection. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 27944, 2021.
[30] Xue Yang, Junchi Yan, Ziming Feng, and Tao He. R3det: Refined single-stage detector with feature refinement for rotating object. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3163-3171, 2021. 2021.
[31] Jiaming Han, Jian Ding, Jie Li, and Gui-Song Xia. Align deep features for oriented object detection. IEEE Transactions on Geoscience and Remote Sensing, 60:1-11, 2022.
[32] Xue Yang, Jirui Yang, Junchi Yan, Yue Zhang, Tengfei Zhang, Zhi Guo, Xian Sun, and Kun Fu, Scrdet: Towards more robust detection for small, cluttered and rotated objects. In JEEL/CVF International Conference on Computer Vision, pages 8231-8240, 2019.
[33] Wen Qian, Xue Yang, Silong Peng, Junchi Yan, and Yue Guo. Learning modulated loss for rotated object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2458-2466, 2021.
[34] Xue Yang and Junchi Yan. Arbitrary-oriented object detection with circular smooth label. In European Conference on Computer Vision, pages 677-6677-699 7644, 2020.
[35] Xue Yang, Liping Hou, Yue Zhou, Wentao Wang, and Junchi Yan. Dense label encoding for boundary discontinuity free rotation detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15814-15824, 2021.
[36] Yi Yu and Feipeng Da. Phase-shifting coder: Predicting accurate orientation in oriented object detection. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
12

---

[37] Xue Yang, Junchi Yan, Ming Qi. Wentao Wang, Xiaopeng Zhang, and Tian Qi. Rethinking rotated object detection with gaussian wasserstein distance loss. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 11830-11841, 2021.
[38] Xue Yang, Xiaojiang Yang. Jirui Yang. Qi Ming. Wentao Wang. Qi Tian, and Junchi Yan. Learning high-precision bounding box for rotated object detection via bullback-leibles divergence. Advances in Neural Information Processing Systems, volume 34, pages 18381-18399, 2021.
[39] Xue Yang. Gefan Zhang, Xiaojiang Yang. Yue Zhou, Wentao Wang, Jin Tang, Tao He, and Junchi Yan. Detecting rotated objects as gaussian distributions and its 3-d generalization. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(4):4335-4354, 2023.
[40] Xue Yang, Yue Zhou, Gefan Zhang, Jirui Yang. Wentao Wang, Junchi Yan, Xiaopeng Zhang, and Qi Tian. The kfiou loss for rotated object detection. In International Conference on Learning Representations, 2023.
[41] Yang, Shaohui Liu, Han Hu, Liwei Wang, and Stephen Lin. Reppoints: Point set representation for object detection, In IEEF/CVF International Conference on Computer Vision, pages 9656-9665, 2019.
[42] Liping Hou, Ke Lu, Xue Yang, Yuqiu Li, and Jian Xue. G-rep: Gaussian representation for arbitrary- oriented object detection, arXiv preprint arXiv:2205. 11796, 2022.
[43] Wentong Li, Yijie Chen, Kaixuan Hu, and Jianke Zhu. Oriented reppoints for acrial object detection. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1829-1838, 2022.
[44] Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, and Bernt Schiele. Simple does it: Weakly supervised instance and semantic segmentation. In JEFE Conference on Computer Vision and Pattern Recognition, pages 876-885. 2017.
[45] Cheng-Chun Hsu, Kuang-Jui Hsu, Chung-Chi Tsai, Yen-Yu Lin, and Yung-Yo Yu Chuang. Weakly supervised instance segmentation using the bounding box tightness prior. Advances in Neural Information Processing Systems, 32, 2019.
[46] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In IEEE International Conference on Computer Vision, pages 2961-2969, 2017.
[47] Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In European Conference on Computer Vision, pages 282 298, 2020.
[48] Javed Iqbal, Muhammad Akhtar Munir, Arif Mahmood, Afsheen Rafagat Ali, and Mohsen Ali. Leveraging orientation for weakly supervised object detection with application to firearm localization. Neurocomputing, 440:310-322 320, 2021.
[49] Yongqing Sun, Jie Ran, Feng Yang, Chenqiang Gao, Takayuki Kurozumi, Hidcaki Kimata, and Ziqi Yc. Oriented object detection for remote sensing images based on weakly supervised learning. In 202/ JEEE International Conference on Multimedia & Expo Workshops (ICMEW), pages 1-6. IEEE, 2021.
[50] Tianyu Zhu, Bryce Ferenezi, Pulak Purkait, Tom Drummond, Hamid Rezatofighi, and Anton van den Hengel. Knowledge combination to learn rotated detection without rotated annotation. /EEF/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[51] Seungkyu Lee and Yanxi Liu. Curved glide-reflection symmetry detection. IEEE Conference on Computer Vision and Pattern Recognition, pages 1046-1053, 2009.
[52] Tianshu YU, Junchi Yan, Jieyi Zhao, and Baochun Li. Joint cuts and matching of partitions in one graph. In ILEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[53] Rajendra Nagar and Shanmuganathan Raman. Detecting approximate reflection symmetry a point set
using optimization on manifold. IEEE Transactions on Signal Processing, 67(6):1582-15 1595, 2019.
[54] Rajendra Nagar and Shanmuganathan Raman. 3dsymm: Robust and accurate 3d reflection symmetry
detection. Pattern Recognition, 107:107483, 2020.
[55] Ahyun Seo, Byungjin Kim, Suha Kwak, and Minsu Cho. Reflection and rotation symmetry detection via equivariant learning. In IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages 9529-9538, 2022.
[56] Yifei Shi, Xin Xu, Junhua Xi. Xiaochang Hu, Dewen Hu, and Kai Xu. Learning to detect 3d symmetry from single-view rgb-d images with weak supervision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4884):4882 4896, 2023.
13

---

[57] Yanxi Liu. Computational symmetry. In Computer Vision: A Reference Guide. Springer International Publishing, 2020.
[58]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
[59] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In IEEE Conference on Computer Vision and Puttern Recognition, pages 936 944, 2017.
[60] Jiahui Yu, Yuning Jiang, Zhangyang Wang, Zhimin Cao, and Thomas Huang. Unitbox: An advanced object detection network. In Proceedings of the 24th ACM International Conference on Multimedia, pages 516-520, 2016.
[61] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performanee deep learning library. In Advances in Neural Information Processing Systems, volume 32, pages 8024-8035 2019.
[62] Yue Zhou, Xue Yang, Gefan Zhang, Jiabao Wang. Yanyi Liu, Liping Hou, Xue Jiang. Xingzhao Liu, Junchi Yan, Chengqi Lyu, Wenwei Zhang, and Kai Chen. Mmrotate: rotated object detection benchmark using pytorch. Proceedings of the 30th ACM International Conference on Multimedia, 2022.
[63] Liping Hou, Ke Lu, Jian Xue, and Yuqiu Li. detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 923 -322, 2022.
selection and measurement for oriented object
[64] Zonghao Guo, Chang Liu, Xiaosong Zhang, Jianhin Jiao, Xiangyang Ji, and Qixiang Yc. Beyond bounding- box: Convex-hull feature adaptation for oriented and densely packed object detection. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8788-8797, 2021.
[65] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEF/CVF International Conference on Computer Vision, pages 9992-10002, 2021.
[66] Xian Sun, Peijin Wang, Zhiyuan Yan, Feng Xu, Ruiping Wang, Wenhui Diao, Jin Chen, Jihao Li, Yingchao Feng, Tao Xu, Martin Weinmann, Stefan Hinz. Cheng Wang, and Kun Fu. Fairlm: A benchmark dataset fine-grained object recognition in high-resolution remote sensing imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 184:1 16-130, 2022.
[67] Ilya Loshchilov Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Reprexentations, 2018
[68] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
14


Reviews:
[{"review_id": "JJlcheC1tF", "reviewer": "Reviewer_k1cY", "summary": "This paper proposes a new horizontal box-supervised rotation object detection detector. The proposed detector consists of two modules: a self-supervised regression branch for angle regression and a weakly supervised branch for horizontal box regression. This method is more simpler and shows clear improvement over the previous one.", "questions": "- I have a question about the self-supervised branch in the proposed method. Why was it designed with two perspectives instead of using one perspective and randomly selecting multiple variations under that perspective? Additionally, would using a more perspective, such as different rotation angles, combinations of rotation and symmetries, or scaling variations, lead to performance improvements?\n\n- Regarding Table 4, I noticed that the experiments on DOTA and HRSC datasets show inconsistent results. Specifically, in the Dota dataset, not using PSC will result in an incorrect result, while in the HRSC dataset, not using PSC only results in a performance loss. This raises the question of whether PSC only improves performance or if it has an indispensable impact on the model.", "limitations": "\nN/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper fully considers the independence and correlation between angle and horizontal box in rotation object detection. It only uses angle regression in self-supervised regression while using Circumscribed RBox IoU to associate angle regression and horizontal box regression in weakly supervised regression to obtain the rotated box. \n-  The paper greatly improves the performance of the hbox-supervised detector, approaching or even reaching the performance of some rbox-supervised detectors.\n- The paper is well written, and the method is clear and well described.", "weaknesses": "- Figure 2 seems a bit complicated, it would be better to highlight the main points to make it clearer.\n- Although this article has achieved impressive results, it is important to consider the robustness of the model when horizontal bounding box annotations are not accurate enough, the article's experiments do not explicitly demonstrate the robustness towards inaccuracies in horizontal bounding box annotations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Review k1cY\n\nThank you for the nice comments and valuable suggestions. By revising accordingly, the article is now clearer and more complete!\n\n**Q1 Robustness to inaccuracies in horizontal bounding box annotations**\n\nThanks. We add some random noise to the annotation and record AP50/AP75 under different noise levels on DOTA-v1.0. Noise=30% indecates Height=Height*[0.7-1.3] and Width=Width*[0.7-1.3], both from uniform distribution. The results are as follows:\n\n| Noise     |      0%     |     10%     |     30%     |     50%     |\n|:---------:|:-----------:|:-----------:|:-----------:|:-----------:|\n| H2RBox    | 70.05/38.38 | 69.19/35.24 | 67.39/26.02 | 61.66/14.55 |\n| H2RBox-v2 | 72.31/39.49 | 71.68/36.33 | 71.11/34.12 | 67.88/21.56 |\n\nResults show that when adding 30% random noise to the annotations, the AP50 of H2RBox-v2 drops by only 1.2%, less than H2RBox (2.69%), which demonstrates the better robustness of our method. We will also add these results in the new version to make it more convincing.\n\n**Q2.a About the self-supervised branch**\n\nThank you for bringing up this interesting idea. Our two-perspective paradigm is intuitively derived from our theory which involves two consistencies, and you provide a new multiplexing solution for symmetry-aware learning. According to your suggestion, we conduct two additional experiments:\n\n1. Randomly select from 5% flip or 95% rotation in one perspective/branch. \n2. Use one more perspective/branch with another random rotation angle.\n\nThe results (AP50/AP75) are as follows:\n\n| Exp ID    |   original  |rand sel.    | +branch     |\n|:---------:|:-----------:|:-----------:|:-----------:|\n| H2RBox-v2 | 72.31/39.49 | 72.24/39.51 | 72.05/39.18 |\n\nThe results show that the multiplexing setting reaches almost the same accuracy as the original one. We will also add this result in the new version.\n\n**Q2.b Using scaling variations**\n\nDOTA usually follows two protocols: with and without multi-scale (MS). In the \"w/o MS\" setting, using scaling can improve the performance, but it also makes the comparison unfair if we integrate scaling and compare with baselines without MS. Whereas, in the \"w/ MS\" setting, reflection/scaling are already used as augmentation. In this case, we suppose using them in view generation plays the same role as in augmentation, on the ground that scaling does not change the angle, and the consistency loss between the original view and the scaling one is zero. We will further explore your suggestions in future work.\n\n**Q3 Inconsistent results in Table 4. Does PSC have an indispensable impact?**\n\nPSC is indispensable for the stability of our model. \n\nWithout PSC, we had empirically observed that the loss could fluctuate in a wide range (possibly due to the angular boundary discontinuity), even failure in convergence. In comparison, when both PSC and snap loss are used, the training is very stable, with not a single failure in our entire experiments. \n\nIn terms of your mentioned results, they seem to be inconsistent, but the instability is consistent, and both results prove the necessity of PSC and snap loss. \n\nPlease let us know if there are further questions."}, {"review_id": "Vr1zPlCsDv", "reviewer": "Reviewer_W3Ay", "summary": "This paper introduces H2RBox-v2, an innovative approach to further bridge the gap between HBox-supervised and RBox-supervised oriented object detection. It seeks to address the limitations of the original H2RBox model, which required high-quality annotations and large training datasets, and was incompatible with rotation augmentation. H2RBox-v2 augments the original model by adding a self-supervised branch that learns object orientations from inherent visual symmetry, and a weakly-supervised branch that incorporates a new CircumIoU loss to allow for random rotation augmentation.", "questions": "- Would it be more effective to conduct the comparative experiments on the DOTA dataset, regarding the assertion that H2Rbox-v1 requires more training data than v2? Specifically, by training both v1 and v2 with varying percentages of DOTA data (namely 10%, 20%, and 50%), we could gain a more robust comparison of their respective performances.\n\n- When the object is not in the center of the image, are Way1 and Way2 described in L124-132 still equivalent?\n\n- In remote sensing scenarios, some objects may not possess symmetry, such as swimming pools and harbors. What impact does this have on the theory?", "limitations": "Nothing to report.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Originality:** While the paper builds upon existing work (specifically H2RBox), it infuses new concepts, the most significant being the utilization of symmetry for angle regression. This concept is highly innovative and introduces a novel angle of approach for oriented object detection. This creativity in applying the natural property of symmetry to enhance detection accuracy sets this work apart from previous methods and broadens the boundary of the field.\n\n- **Quality:** The proposed H2RBox-v2 exhibits enhanced performance on various datasets compared to its predecessor, extensive ablation experiments have demonstrated the importance of each module.\n\n- **Clarity:** The paper is well-structured and clearly presents the methodology and results, making the contributions of this research easily understandable.\n\n- **Significance:** Bridging the gap between HBox-supervised and RBox-supervised oriented object detection significantly reduce the annotation costs.\n\n- H2RBox-v2 has shown improved performance on various datasets and is specifically designed to cope with situations where H2RBox-v1 may underperform. This makes it a valuable contribution to real-world applications of oriented object detection.", "weaknesses": "- **Theory is not fully comprehensive:** *L138-142* only discussed the case where a single instance is contained in an image. However, in reality, remote sensing images can contain many objects, especially in dense scenarios. Despite the theoretical proof being not fully comprehensive, their simplicity of idea and empirical effectiveness seem to be sufficient according to me.\n\n- **Training overhead not detailed:** It is crucial that the model does not introduce additional overhead during the testing phase, but the paper does not discuss the computational overhead and the time taken for the model training process in detail", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer W3Ay\n\nThanks for your positive comments and constructive suggestions. Your endorsement of our method gives us significant encouragement.\n\n**Q1 The computational overhead and the time taken for the model training/testing process**\n\n**Test phase:** In our submission, we only gave FPS for evaluating inference time (in Table 2). We now add the accurate computational costs (the additional cost of -v2 is due to PSC decoding) and we will also enrich the experiment section in our new version:\n\n|        |  H2RBox (v1)  |   H2RBox-v2   |\n|:------:|:-------------:|:-------------:|\n| Flops  | 206.91 GFLOPs | 207.01 GFLOPs |\n| Params |    31.92 M    |    31.93 M    |\n\n**Train phase:** H2RBox-v2 is slower than -v1 in training as it involves one more branch. Here we provide an additional experiment that randomly selects from 5% flip or 95% rotation in only one branch (\"5%\" is based on $\\lambda=0.05$ in Table 6). The resulting AP50/AP75 are:\n\n|  Methods  | original (2 branches) | multiplex (1 branch) |\n|:---------:|:---------------------:|:--------------------:|\n| H2RBox-v2 |      72.31/39.49      |     72.24/39.51      |\n\nThe multiplex version requires similar training time to -v1 while keeping high performance as -v2:\n\n| FCOS  | H2RBox-v1 | H2RBox-v2 | H2RBox-v2 (multiplex) |\n|:-----:|:---------:|:---------:|:---------------------:|\n| 5h10m |   7h10m   |   8h56m   |         7h7m          |\n\n**Q2 Comparative experiments on the varying percentages of DOTA dataset, regarding the assertion that H2Rbox-v1 requires more training data than v2**\n\nThanks for your nice suggestions. In the following table, we show that the gap between -v1 and -v2 becomes larger on the sampled version of DOTA dataset (30%, 10%). The AP50/AP75 are:\n\n|  Methods  |     Full    |     30%     |     10%     |\n|:---------:|:-----------:|:-----------:|:-----------:|\n| H2RBox    | 70.05/38.38 | 55.73/20.14 | 37.71/ 6.98 |\n| H2RBox-v2 | 72.31/39.49 | 61.25/27.91 | 44.61/14.97 |\n\n**Q3 When the object is not in the center of the image, are Way1 and Way2 still equivalent?**\n\nYes, actually for any image and any $\\theta$, { flip about line $\\theta$ } is equivalent to { flip vertically and then rotate by $2 \\theta$ }. But when the object is not in the center, the input image becomes asymmetric, so $f_\\text{nn}\\left ( I' \\right ) = f_\\text{nn}\\left ( I_0 \\right )$ (on Line 125) does not hold. \n\nTechnically speaking, when the object is not in the center, the situation is similar to multiple object detection. Although our theoretical study is performed on a single object, there is an assigner in the SS branch to match the center of objects in different views (see Line 151), and the consistency loss is calculated between these matched center points. With the assigner, our \"single-object\" theory can be applied to each matched object center, and this is the way our network can be used for multiple objects (and objects not in the center).\n\n**Q4 Some objects may not possess symmetry, e.g. swimming pools and harbors. What impact does this have on the theory?**\n\nWhile the training objects are preferred to be symmetric which often holds in aerial images, our experiments show that the symmetry need not be strictly obeyed. H2RBox-v2 can still optimize for the most likely solution -- an approximate axis that divides the object into two \"most mirrored\" parts. This mechanism extends the applicability of H2RBox-v2 to most elongated objects. As a result, the performance of each class and the visualization in our supplementary material demonstrates that H2RBox-v2 still gives a competitive performance in terms of swimming pools and harbors. \n\nPlease let us know if there are further questions."}, {"review_id": "Ba1NzQ4S5r", "reviewer": "Reviewer_tfHb", "summary": "This paper proposes an advance solution for using horizontal bounding boxes as supervision to learn oriented object detectors. \nThe proposed method H2RBox-V2, which is a modification of the recent work H2RBox, has some technique novelty and contribution as it jointly uses the weakly- and self-supervised branches to learn the rotation angle. \nExperiments are validated on DOTA, HRSC and FAIR1M, and show state-of-the-art performance.\n", "questions": "\nThe authors have addressed my questions in the rebuttal well.\n\n\n______ before rebuttal _____________\n\n\nQ1: Issues on the flaw of the technique framework. Both experimentally and theoretically.\n\nQ2: The details, design and clear presentation of the CircumIoU loss.\n\nQ3: systematically address the unfair experimental comparison issue. \n", "limitations": "The authors have addressed my concerns well.\n\n______________________before rebuttal ___________\nThe limitations are not properly discussed. The reviewer believes the technique flaws as mentioned in weakness part is more critical to the proposed framework. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The overall idea to learn the rotated angles from both weakly- and self- supervised branch is very interesting.\n\n+ The task itself, to learn rotated bounding boxes from horizontal bounding boxes, is a new setting for the rotated object detection community. \n\n+ The performance improvement against prior arts is significant.\n", "weaknesses": "\n- Unclear methodology design and presentation.\n\n(1) The authors claim to propose a new CircumIoU loss for this framework. Unfortunately, in the methodology section, especially Sec3.4, there is no term ‘CircumIoU loss’, and it is impossible to know which loss is the claimed novelty. \n\n(2) Besides, given this unclear description, it is very difficult to know its difference or novelty against some prior works such as [a], which also optimizes the detector from rotation angles.\n\n[a] Arbitrary-Oriented Object Detection with Circular Smooth Label. ECCV 2020.\n\nOther minor issues and comments for improvement:\n\n- Please provide some rotation angle distribution visualizations from the proposed framework.\n\n- Fig.1, column2 and column4. The predictions from the proposed method and SAM-RBox seem not to have much difference. Please consider use more representative figures.\n\n- Table 1, the performance gap. Please explictly mention which dataset it is that leads to 3.41% drop and 0.07% improvement.\n\n- Fig.2 looks very crowded. It can be better polished, and the size of each content can be made more fit.\n\n- For loss function, please use \\mathcal{} to distinguish it from scalars.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer tfHb\n\nThank you for the time and nice suggestions. We humbly point out that there may exist misunderstandings in the review, and we hope our clarification could help on a more informed rating to our work.\n\n**Q1.a Eq. 2 and Eq. 4 conflict: When k is an odd number, the nets learn the opposite rotation angle, i.e. $\\theta$ and $\\theta + \\pi$**\n\nUnfortunately that we didn't well explain or emphasize the basic concept in oriented object detection that bounding boxes have the periodicity of $\\pi$. \n\nAngle $\\theta$ and $\\theta + \\pi$ refer to the same bounding box. Similarly, symmetric axis of $\\theta$ is also equivalent to symmetric axis of $\\theta + \\pi$. We will clarify it in our new version.\n\nBased on our above clarification, Eq. 4 is the extended version of Eq. 2, which takes into consideration the equivalence between symmetric axes of $\\theta + k\\pi$. \n\n**Q1.b Lack experiment: Can using the weakly-supervised branch to predict warrant the same rotation angle as the semi-supervised branch, or at least very close?**\n\nWe hope our above explanation could have eased your concerns and potential misunderstanding. \n\nFor your specific question here, strictly speaking, our method doesn't have a \"semi-supervised branch\", do you mean the self-supervised branch? Actually, there is only one angle prediction during inference. If this misunderstanding is aroused by the multiple networks drawn in Fig. 2, we clarify that these networks/branches share the same parameters (see Line 149 and in Fig. 2). Such a parameter-shared graph is widely used (e.g. also in H2RBox). Therefore, no matter which branch is used for inference, the result should be the same in theory.\n\nWe further use HRSC to compare the AP50 by using different branches for inference:\n\n| WS | SS-Flip | SS-Rotate |\n|:-:|:-:|:-:|\n| 89.66 | 89.66 | 89.66 |\n\nThe above results confirm that with the same input image, all parameter-shared branches have the same output.\n\n**Q2.a The authors claim to propose a new CircumIoU loss, but in the methodology, especially Sec 3.4, there is no term \"CircumIoU loss\"**\n\nThanks for your careful review and suggestion. The term \"CircumIoU loss\" in Sec 3.4 is at Line 182 and in Fig. 4 (b). In Fig. 4 (b), we use a graph similar to the well-known paper \"IoU loss\" to illustrate the calculation process.\n\nWe give a more specific description as follows, which will be added to our new version:\n\nTo calculate CircumIoU loss, the predicted box $B_\\text{pred}$ is first projected to the direction of ground-truth box $B_\\text{gt}$. The obtained projected box $B_\\text{proj}$ is displayed as the dashed box in Fig. 4 (b). Afterward, the loss can be calculated as: \n\n$-\\ln \\frac{intersection(B_\\text{proj}, B_\\text{gt})}{union(B_\\text{proj}, B_\\text{gt})}$\n\nCircumIoU loss can enable H2RBox-v2 to use random rotation (RR) data augmentation to further improve the performance (as shown in Tab. 2), which is not supported by H2RBox-v1.\n\n**Q2.b Difference or novelty against prior works such as Arbitrary-Oriented Object Detection with Circular Smooth Label (ECCV 2020)**\n\nCircular Smooth Label (CSL) is about **supervised** learning that learns angle classification from **angle annotations**. However, as described in our paper's title and introduction, H2RBox-v2 is aimed at a different and more challenging setting -- **self-learning** oriented boxes from **horizontal box annotation** without human labeled angle. CSL cannot even work without labeled angles. Therefore, the difference (novelty) between them is huge.\n\n**Q3.a Prior work H2RBox with 1x schedule and multi-scale (MS) has mAP of 74.40%. But in this submission, H2RBox is made deliberately low by removing its MS scheme. More importantly, all the H2RBox-v2 performance is reported with MS scheme. If consider MS for H2RBox, it can even outperform v2**\n\nThanks for your careful thoughts on our work and giving us the chance to clarify and improve the unclear part of the paper.\n\nPlease note that in fact all the H2RBox-v2 performances in this paper are reported with the **single-scale** scheme unless otherwise specified in Table 2 in the paper, where it shows the performance of **H2RBox-v2 using multi-scale: 77.97%**. Compared to the performance of H2RBox with MS reported in the original paper (74.40%), the improvement is even higher than the value 2.26% that we claimed in our paper (see line 229: H2RBox-v2 outperforms H2RBox by 2.26% = 72.31% - 70.05%). \n\nWe will add the following comparison (i.e. enrich/update Table 2). Instead of \"74.40%\", we will use our reproduced result 75.35% (also a higher baseline).\n\n| Methods | 1x w/o MS | 1x w/ MS |\n|:-:|:-:|:-:|\n| H2RBox (R50) | 70.05% | 75.35% |\n| H2RBox-v2 (R50) | 72.31% | 77.97% |\n| Improvement of v2 | +2.26% | +2.62% |\n\n**Q3.b The highest result of H2RBox-v2 79.75% mAP is made by the Swin-transformer. It is more meaningful to also report the H2RBox (in a fair way) with Swin**\n\nThanks for your suggestion. The new results are as follows (4 GPUs and batchsize=4 to speed up experiments in the limited rebuttal period):\n\n| Methods | 1x w/ MS |\n|:-:|:-:|\n| H2RBox (Swin-Tiny) | 61.60% |\n| H2RBox-v2 (Swin-Tiny) | 79.39% |\n| Improvement of v2 | +17.79% |\n|||\n| H2RBox (Swin-Base) | 61.05% |\n| H2RBox-v2 (Swin-Base) | 80.35% |\n| Improvement of v2 | +19.30% |\n\nOn Swin-transformer, -v2 outperforms -v1 by a large margin. But since the original H2RBox paper does not report performance on Swin, it may be doubted that we \"made low\" the performance of \"H2RBox (Swin)\" for comparison. Therefore, we contacted the authors of H2RBox about the experiment of H2RBox+Swin. They told us that their experimental results are consistent with ours, but they have not found the reason for this phenomenon so far. Anyway, this result demonstrates H2RBox-v2 has more robustness.\n\n**Q4 Other minor issues**\n\nWe will carefully follow these suggestions and revise the paper accordingly. Thanks again for your efforts on this paper!\n\nPlease let us know if there are further questions."}, {"review_id": "lZTbGNHFo7", "reviewer": "Reviewer_Pshp", "summary": "This paper proposes to exploit the reflection symmetry as a new supervision to HBox-supervised oriented object detection. Several modifications are made to adapt the new self-supervised (SS) branch, including removing the angle subnet in the weakly-supervised (WS) branch and a CircumIoU loss for box regression. Experiments show a clear state-of-the-art result to previous Hbox-supervised oriented object detectors.", "questions": "Suggestions:\n\n1. For the weaknesses 1 and 2, I suggest the author to rethink the logic of the analysis of Sec. 3.1. It would be better if we could restate the analysis in page 5 line 130 to the converse proposition - \"If the network $f_{nn}(\\cdot)$ can always predict the symmetrical axis $\\theta_{sym}$ of the image, then, it must subject to both flip and rotate consistencies.\" And then the contrapositive - \"if the network subjects to neither flip consistency nor rotate consistency, it cannot predict the symmetrical axis of the image.\" Therefore, we need to enhance both flip consistency and rotate consistency. H2Rbox considers the rotate consistency only, which leads to sub-optimal results.\n\n2. There is a missing experiment, i.e., $\\lambda=0$ in Tab. 6.", "limitations": "see Weaknesses", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tries to exploit the reflection symmetry to improve the Hbox-supervised object detector, which is interesting.\n\n2. The paper is well-motivated and easy to understand.\n\n3. The proposed H2Rbox-v2 achieves state-of-the-art performance in the Hbox-supervised oriented object detectors.", "weaknesses": "Major concern:\n\n1. The soundness of using reflection symmetry is somewhat low. According to page 5 line 130, if the network $f_{nn}(\\cdot)$ subjects to both flip and rotate consistencies, then $f_{nn}(I_0)=\\theta_{sym}$. I don't know why this equation holds. Clearly, we don't know the symmetrical axis of the image $I_0$. And if we rotate the image by a random angle $\\theta$ in the Way2 of Fig.3, then $f_{nn}(I_2)$ would never be equal to $\\theta_{sym}$. Even if the symmetrical axis is known, this reflection symmetry learning can only theoretically and ideally work under the setting of a single symmetric object. As a matter of fact, it would be more rational that the converse proposition holds - \"If the network $f_{nn}(\\cdot)$ can always predict the symmetrical axis $\\theta_{sym}$ of the image, then, it must subject to both flip and rotate consistencies.\" I suggest the authors reconsider the logic here.\n\n2. Due to weakness 1, the novelty is also limited since one can easily conclude that H2Rbox-v2 is to add one more additional branch with a new view generation upon H2Rbox, i.e., the left column of Fig. 2(b) (\"vertical flipping\"). The right column is random rotation which is similar to the SS branch of H2Rbox. This is what I believe the model brings us performance gains, but not the so-called reflection symmetry learning.  The experiments cannot support that the reflection symmetry is important, but can only verify the effectiveness of adding the vertical flipping branch, which is supported by Tab. 6. It is still unclear for the relationship between reflection symmetry learning and multi-branch self-supervised learning.\n\n3. The ablation studies provide a weak explanation for the effectiveness of the proposed components. In Tab. 4, adding the $l_s$ loss only produces near zero mAP on DOTA, while it is much higher on HRSC. Why did such drastic fluctuations occur? The PSC coder is just an angle encoder-decoder module, which is originally proposed to improve the performance of oriented object detectors. It would not be a critical factor affecting the performance, as demonstrated by the PSC paper, ~2 mAP at most. This paper shows PSC is crucial in the proposed method. That is to say, without PSC, the proposed method produces much poor performance. This largely weakens the soundness of the designs in the SS branch. Additionally, PSC imposes the loss on two encoded phase-shifting patterns of angles, which inherently solves the boundary discontinuity, while the proposed SS branch decodes the phase-shifting patterns to angle and then calculates the loss on two angles, which introduces the boundary discontinuity again, thus undermining the contribution of the snap loss.\n\n4. It is confusing to me that Fig.2 shows the angle is predicted by the SS branch, while all the other properties are predicted by the WS branch. That is to say, the inference time of H2Rbox-v2 (WS+SS) should be double of the WS branch, while H2Rbox only needs the WS branch for inference. I'm concerned about why the FPS of H2Rbox-v2 is equal to H2Rbox as shown in Tab. 2. The authors need to describe the inference process clearly.\n\nMinor concern:\n\nThe baseline model is unclear in the ablation studies. In Tab. 4, if the baseline model is \"w/o PSC and $l_s$\", then what is the angle coder used in the baseline? And what are the angles in calculating Smooth-L1 loss? The same question emerges in Tab. 5 and Tab. 6. Is that mean all the baseline models of the tables adopt the optimal strategies of the other tables? If so, then the best results of Tab.6 seem to be not in line with the others.\n\nOverall:\n\nWhile the proposed method achieves state-of-the-art performance, my initial rating score is 4. I have some concerns about the soundness of the reflection symmetry learning and the ablation studies. It is of vital importance to give an explanation and analysis of the rationality of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer Pshp\n\nWe sincerely appreciate your valuable suggestions. We hope our clarification could help to make a more informed rating to our work.\n\n**Q1 It would be more rational that the converse proposition holds for symmetric axis prediction and flip/rotate consistency**\n\nWe think our (empirically verified) idea is beyond this converse proposition. We restate the main facts and our theory in H2RBox-v2 as: \n\n- **Observation (facts):** Collect a set of images containing a symmetric object along an arbitrary angle, and use this set to train a three-branch like neural network supervised just by the flip and rotate consistencies respectively via the two of the branches **(without the supervision of any annotation)**. We show that the trained network is able to estimate the angle of the symmetric axis of the object.\n\n- **Underlying theory:** We have proved in our paper (see Sec 3.1) that if a function always satisfies the flip and rotate consistencies for the input symmetric image, then the function's output (by certain derivation - see details in the paper) is exactly the angle of the axis of the symmetric image.\n\n- **Handling multiple objects in images:** The above theory strictly holds in the single object case. With an assigner in the SS branch to match objects in different views (see Line 151), the consistency loss is calculated between these matches, so that our \"single-object\" theory can be applied to each matched object.\n\n**Q2.a One more branch is the key to performance improvement rather than reflection symmetry learning**\n\n- **V2 has completely different angle information acquisition mechanism from v1:** Refer to the general response.\n\n- **Performance improvement is not due to one more branch:** A new experiment that randomly selects from 5% flip or 95% rotation in only one branch (based on $\\lambda=0.05$ in Table 6) shows that the number of branches is not the key to improving performance, AP50/AP75:\n\n| Methods | original (2 branches) | multiplex (1 branch) |\n|:-:|:-:|:-:|\n| H2RBox-v2 | 72.31/39.49 | 72.24/39.51 |\n\n**Q2.b Experiments cannot support reflection symmetry is important**\n\nSorry for leaving an impression of \"adding flip to H2RBox-v1\" which is in fact not true. Essentially we discard the use of geometric constraints from human annotations in obtaining angles (the way of v1), and the angles are fully learned by the reflection symmetry of objects (refer to the answer of Q2.a). We are not proving that symmetry improves -v1, yet rather that symmetry can replace the way of learning angle in v1. \n\n**Q2.c Relation between reflection symmetry learning and multi-branch SSL**\n\nThe former is our theory that allows the network to perceive the angle of objects from their symmetry and the latter is a neural network implementation.\n\n**Q3.a The loss (Tab. 4) produces near 0 mAP on DOTA but much higher on HRSC. Why did such drastic fluctuations occur?**\n\nWithout handling boundary discontinuity, we empirically found that the loss could fluctuate in a wide range, even failure in convergence. In comparison, when both PSC and snap loss are used, the training is very stable. The \"drastic fluctuations\" exactly show the instability and prove the necessity of PSC and snap loss. We now give more discussion in the new version.\n\n**Q3.b Poor performance w/o PSC: PSC would not be a critical factor, ~2 mAP at most**\n\nWe agree that the impact of boundary problems may be ~2 mAP in **supervised** setting. Yet the impact of boundary problem in our consistency-based self-supervised setting could be much greater than that in the supervised case, especially in terms of stability, as shown in Table 4. Our point is that, \"the poor performance without PSC\" suggests in fact H2RBox-v2 allows PSC to well address the boundary problem to avoid poor performance. PSC is indispensable in our approach - but it does not mean our other parts' design is not sound. \n\n**Q3.c Calculating the loss on two decoded angles introduces the boundary discontinuity again**\n\nThe snap loss limits the difference between two angles into the $\\pi/2$ (see Fig. 4a), thus the calculation won't introduce the boundary discontinuity again. This is why both PSC and snap loss are important to solve boundary problem in the self-supervised design.\n\n**Q4 H2Rbox-v2 predicts angle by the SS branch, while H2Rbox only needs the WS branch for inference. Why is the FPS equal?**\n\nAs shown in the WS branch in Fig. 2, due to parameter sharing, the inference is in the form of WS + Angle head from SS, so the efficiency is almost the same as that of H2Rbox.\n\nTo be precise, the only additional cost of v2 during inference is PSC decoding. Thus, FCOS/H2RBox-v1/H2RBox-v2 have similar inference times. With input shape (3, 1024, 1024), the accurate costs are:\n\n| | H2RBox | H2RBox-v2 |\n|:-:|:-:|:-:|\n| Flops | 206.91 GFLOPs | 207.01 GFLOPs |\n| Params | 31.92 M | 31.93 M |\n\n**Q5.a When \"w/o PSC and $l_s$\", what is used in the baseline?**\n\n\"w/o PSC\" means that the conv layer directly outputs the angle. \"w/o $l_s$\" means using smooth-L1 loss. We will add the description in our new version.\n\n**Q5.b Do all the tables adopt the optimal strategies of the other tables? Tab.6 seems to be not in line with the others**\n\nYes, we adopt the optimal strategies if not otherwise specified. The best results are not in line with the others because we run the same config ($\\lambda=0.05$) more than once, i.e. DOTA: 40.39/72.59.39.18 (Tab. 6) vs. 40.69/72.31/39.49 (other Tabs.), HRSC: 56.76/89.63/62.93 (Tab. 6) vs. 58.03/89.66/64.8 (other Tabs.), and we forgot to modify the Tab. 6 to the final adopted result. We will unify this result in the final version.\n\n**Q5.c Missing $\\lambda=0$ in Tab. 6**\n\nThe experiment is now added, which shows that the flip branch is necessary (AP50/AP75/AP):\n\n| $\\lambda$ | HRSC | DOTA |\n|:-:|:-:|:-:|\n| 0 | 0.32/0.00/0.06 | 66.37/25.03/31.60 |\n\nPlease let us know if there are further questions."}]

Author rebuttal:
General Response:

We thank the reviewers for their time and constructive suggestions. And the reviewers give appreication in a few points:

1. writing/presentation (**k1cY**:The paper is well written, and the method is clear and well described; **W3Ay**: The paper is well-structured and clearly presented; **Pshp**: The paper is easy to understand)

2. motivation/methodology (**tfHb**: The idea to learn the angles is very interesting; **Pshp**: Exploiting the reflection symmetry to improve the object detector is interesting and the paper is well-motivated; **W3Ay**: The paper infuses new concepts -- utilizing symmetry for angle regression, which is highly innovative; **k1cY**: The paper fully considers the independence and correlation between angle and horizontal box, and it only learns angles in self-supervised regression)

3. experiments/results (**k1cY**: The paper greatly improves the performance, approaching or even reaching the performance of some rbox-supervised detectors; **W3Ay**: It exhibits enhanced performance on various datasets and extensive ablation experiments have demonstrated the importance of each module; **Pshp**: The proposed H2Rbox-v2 achieves state-of-the-art performance in the Hbox-supervised oriented object detectors)

However, there are also some major concerns as follows, in which we have to humbly suggest that there may exist misunderstandings.

Q1. **Unfair comparison** (**tfHb**: Comparison with prior work H2RBox is unfair. All the H2RBox-v2 performance is reported with the multi-scale scheme, but H2RBox is made deliberately low by removing its multi-scale scheme)

Our response: All the H2RBox-v2 performances are reported with the **single-scale** scheme unless otherwise specified in Table 2 in the paper. In our comparison (see line 229: H2RBox-v2 outperforms H2RBox by 2.26% = 72.31% - 70.05%), both sides are based on single-scale. When comparing on multi-scale scheme (according to the reviewer's suggestion), the improvement is 2.62%, even higher than the value 2.26% that we claimed.

Q2. **Soundness of the theory** (**tfHb**: Eq. 2 and Eq. 4 conflict: When k is an odd number, the nets learn the opposite rotation angle, i.e. $\theta$ and $\theta + \pi$; **Pshp**: I don't know why the equation $f_\text{nn}\left ( I_0 \right ) = \theta_\text{sym}$ holds)

Our response:

**to tfHb:** In oriented object detection, bounding boxes have the periodicity of $\pi$. Angle $\theta$ and $\theta + \pi$ refer to the same bounding box. Similarly, symmetric axis of $\theta$ is also equivalent to symmetric axis of $\theta + \pi$. 

**to Pshp:** The equation is solved from $f_\text{nn}\left ( I_0 \right ) = \theta_\text{pred} = - \theta_\text{pred} + 2\theta_\text{sym}$, which is mathematically derived from the equivalence between Way 1 and Way 2 (see definition of the two ways in Line 124-127).

Q3. **Improvement not due to the new theory** (**Pshp**: One more branch than H2RBox (v1) is the key to performance improvement rather than so-called reflection symmetry learning)

Our response: H2RBox-v2 has a completely different angle information acquisition mechanism from v1. To prove v2 learns angle from the image (unlike v1 from annotation), we provide an additional experiment (see the attached PDF in this rebuttal). We weaken the annotation of DOTA to square boxes so that they don't contain any angle-related information. The results show that in this experiment v1 fails to learn the correct angle, while v2 still finds the correct angle. This verifies the soundness of our theory (If our theory doesn't work, v2 should be the same as v1 in this experiment).

We hope our clarification could help to make a more informed rating to our work. In the following individual response, we provide answers to each raised weakness/question.

Best regards,

Authors

Meta-review: The paper proposes an improved version of H2RBox for oriented bounding box detection. It is an interesting but infrequently discussed question in the literature, while all the reviewers are positive about the paper after the rebuttal. The AC recommends accepting the paper.
Decision: Accept (poster)