Title: Progressive Exploration-Conformal Learning for Sparsely Annotated Object Detection in Aerial Images
Venue: NeurIPS 2024
Accepted: yes
Keywords: Aerial object detection; Sparse annotation; Conformal exploratory learning
Abstract: The ability to detect aerial objects with limited annotation is pivotal to the development of real-world aerial intelligence systems. In this work, we focus on a demanding but practical sparsely annotated object detection (SAOD) in aerial images, which encompasses a wider variety of aerial scenes with the same number of annotated objects. Although most existing SAOD methods rely on fixed thresholding to filter pseudo-labels for enhancing detector performance, adapting to aerial objects proves challenging due to the imbalanced probabilities/confidences associated with predicted aerial objects. To address this problem, we propose a novel Progressive Exploration-Conformal Learning (PECL) framework to address the SAOD task, which can adaptively perform the selection of high-quality pseudo-labels in aerial images. Specifically, the pseudo-label exploration can be formulated as a decision-making paradigm by adopting a conformal pseudo-label explorer and a multi-clue selection evaluator. The conformal pseudo-label explorer learns an adaptive policy by maximizing the cumulative reward, which can decide how to select these high-quality candidates by leveraging their essential characteristics and inter-instance contextual information. The multi-clue selection evaluator is designed to evaluate the explorer-guided pseudo-label selections by providing an instructive feedback for policy optimization. Finally, the explored pseudo-labels can be adopted to guide the optimization of aerial object detector in a closed-looping progressive fashion. Comprehensive evaluations on two public datasets demonstrate the superiority of our PECL when compared with other state-of-the-art methods in the sparsely annotated aerial object detection task.

Progressive Exploration-Conformal Learning for Sparsely Annotated Object Detection in Aerial Images

Abstract

The ability to detect aerial objects with limited annotation is pivotal to the development of real-world aerial intelligence systems. In this work, we focus on a demanding but practical sparsely annotated object detection (SAOD) in aerial images, which encompasses a wider variety of aerial scenes with the same number of annotated objects. Although most existing SAOD methods rely on fixed thresholding to filter pseudo-labels for enhancing detector performance, adapting to aerial objects proves challenging due to the imbalanced probabilities/confidences associated with predicted aerial objects. To address this problem, we propose a novel Progressive Exploration-Conformal Learning (PECL) framework to address the SAOD task, which can adaptively perform the selection of high-quality pseudo-labels in aerial images. Specifically, the pseudo-label exploration can be formulated as a decision-making paradigm by adopting a conformal pseudo-label explorer and a multi-clue selection evaluator. The conformal pseudo-label explorer learns an adaptive policy by maximizing the cumulative reward, which can decide how to select these high-quality candidates by leveraging their essential characteristics and inter-instance contextual information. The multi-clue selection evaluator is designed to evaluate the explorer-guided pseudo-label selections by providing an instructive feedback for policy optimization. Finally, the explored pseudo-labels can be adopted to guide the optimization of aerial object detector in a closed-loop progressive fashion. Comprehensive evaluations on two public datasets demonstrate the superiority of our PECL when compared with other state-of-the-art methods in the sparsely annotated aerial object detection task. The code will be available at: https://github.com/SAOD-research/PECL.

1 Introduction

Recently, object detection has gained widespread attention, but the demand for a large amount of labeled data is time-consuming and labor-intensive. To address this challenge, semi-supervised object detection (SSOD) has been proposed to enhance detection performance by utilizing limited labeled samples and a large number of unlabeled samples. SSOD methods (14; 30; 23; 36) mainly focus on general images, and cannot consider the unique characteristics of objects in aerial images, such as dense arrangements, rich contextual relationships, and large complex scenes. As shown in Figure 1(a), objects in aerial images are more dense and similar compared to general images. For example, the average objects per image are 68.4 vs 7.7 in the DOTA (27) and COCO (13) datasets, respectively. Therefore, we address the sparsely annotated object detection (SAOD) task in aerial images, which

(a) the global aerial image

(b) the local sparsely-annotated objects

labels a few part of objects in the training set. Compared with the conventional SSOD task, the SAOD can cover more diversity of aerial scenes even with the same number of annotated objects, as illustrated in Figure 1(b).

The SAOD task mitigates the need for expensive instance-level annotations, but also faces severe challenges of detector optimization, especially only with the limited and sparse annotation objects. In general, most existing SAOD approaches have attempted to mine more confidence supervised signals (e.g., pseudo-labels) from a substantial quantity of unlabeled aerial images. For example, Yoon et al. (32) combined an anchor-free object detector with an object tracker to generate dense annotations for training images. Co-mining (26) employed a Siamese network comprising two branches that mutually predict pseudo-label sets. Niitani et al. (18) utilized a part-aware sampling technique to handle instances within primary categories and used pseudo-labels to exclude un-annotated regions. However, existing SAOD methods typically focus on using fixed thresholds to filter pseudo-labels, lacking the ability to adaptively select high-quality pseudo-labels. Different from conventional objects, some aerial objects characterized by substantial dimensions and notable features (e.g., planes) exhibit higher predicted probabilities, whereas smaller aerial objects like vehicles are associated with comparatively lower predicted probabilities. Consequently, the discrepancy in probabilities and confidence levels observed in the predicted results poses a severe challenge in identifying pseudo-label instances by setting a rigorous threshold in the sparsely annotated aerial object detection process. Meanwhile, the conformal prediction methodology, as elucidated by Shafer et al. (2008) (20), emerges as a potent approach for quantifying uncertainty by harnessing confidence levels to effectively rectify the challenges stemming from imbalanced data.

In this work, we propose a Progressive Exploration-Conformable Learning (PECL) framework to improve the SAOD performance in aerial images, which can adaptively explore more confident pseudo-labels by considering the imbalance characteristics of different categories. Specifically, we firstly pre-train the detector with these sparsely annotated objects, and employ online clustering to generate class-wise knowledge to assist the pseudo-label exploration. The conformal decision problem of pseudo-label exploration is addressed with two essential components: the conformal pseudo-label explorer and multi-clue selection evaluator. The conformal pseudo-label explorer is responsible for selecting confident pseudo-labels by considering the intrinsic and conformal characteristics of the predicted candidates; while the multi-clue selection evaluator assesses the pseudo-label exploration and provides feedback to guide the optimization of the pseudo-label explorer. Finally, both the detector updating and the conformal pseudo-label exploration are integrated into a closed-loop, mutually reinforcing framework, promoting the SAOD task.

To summarize, the contributions of this work are threefold: i) We propose a progressive explorationconformable learning framework that integrates the detector updating and the conformal pseudo-label exploration into an iteratively co-enhancing system. ii) We perform the pseudo-label exploration to mine more high-quality pseudo-labels by considering contextual information in large complex scenes, consisting of two modules: the conformal pseudo-label explorer and multi-clue selection evaluator. iii) Comprehensive evaluations on two public datasets, DOTA (27) and HRSC2016 (15), demonstrate the effectiveness of our PECL, which outperforms the baselines and state-of-the-art methods by a large margin in the sparsely annotated aerial object detection task.

2 Related Work

Semi-/weakly-supervised/sparsely-annotated object detection: Due to the extensive annotations requiring a significant investment of time and labor, SSOD and SAOD have attracted considerable attention in recent years. SSOD aims at the detector optimization by using limited labeled samples and a large number of unlabeled samples. For example, Pseudo labeling (10) has been widely used in semi-supervised learning, which utilizes pre-trained detector to generate labels for unlabeled data. Furthermore, some methods (2; 28; 11) combined consistency regularization with pseudo labeling. CSD (9) leveraged consistent predictions of horizontally flipped image pairs, STAC (21) enforced consistency constraints on weakly and strongly augmented image pairs. Unbiased Teacher (14) adopted the focal loss (12) to resolve pseudo-label bias caused by class imbalance in real labels. SOOD (7) designed rotation-aware adaptive weighting loss and global consistency loss for semisupervised oriented object detection. Furthermore, considerable efforts have been made in weaklysupervised oriented bounding box detection. H2RBox (31) learned object center from horizontal box labels in weak-supervision, used scale and spatial constraints to estimate object dimensions in self-supervision. PointOBB (16) and Point2RBox (34) used Multiple Instance Learning and Knowledge Combination to learn rotated box regression from single point supervision. While SAOD labels part of instances in each image. Many researchers have explored methods to address this challenge. Niitani et al. (18) introduced part-aware sampling that leveraged the logical relationship between parts to guide the sampling process. Co-mining (26) involved a Siamese network with two branches that predicted pseudo-label sets for each other. Rambhatla et al. (19) treated SAOD as a semi-supervised problem focusing on regions, automatically identifying unlabeled foreground objects. Calibrated Teacher (25) calibrated the predicted confidence to match the actual accuracy, ensuring detectors at different training stages share similar confidence distributions and fixed threshold. In contrast, our work focuses on sparsely annotated aerial object detection by adaptively exploring contextual relationships between unlabeled instances in large complex scenes.

Reinforcement/exploratory learning in computer vision: Deep reinforcement learning has shown promising results in many decision-making domains. It has also been widely applied in the field of computer vision. Caicedo et al. (3) transformed the object detection into a Markov Decision Process to deform bounding boxes for accurate localization. Huang et al. (8) proposed an adaptive tracking method, which used reinforcement learning to select the number of network layers. DSN (37) treated video summarization as a sequence prediction problem, predicting frame probabilities and selecting frames based on the probability. Yu et al. (33) applied reinforcement learning for image restoration, employing a policy network to select suitable repair tools. Walid et al. (1) formulated the landmark localization in 3D medical images as a reinforcement learning problem and introduced actor-critic for localization tasks. Tian et al. (24) introduced a medical image segmentation method based on reinforcement learning, the reward showed the change between the predictions and the ground truth. IVADC-FDRL (17) empowered the agent to learn from real data by integrating detector for anomaly detection and Q-learning for anomaly classification. In this work, we build a progressive explorationconformal learning process to adaptively select high-quality pseudo-labels, thereby enhancing the supervision signal for sparsely annotated aerial object detection.

3 The Proposed Method

3.1 Problem Definition

The sparsely annotated aerial object detection is to learn a robust aerial detector by employing these sparsely annotated instances and a large amount of unlabeled images/regions in the training set. Formally, we define X = { X i , O i } N i =1 as the training set, where N represents the total number of training images, X i and O i denote the i -th image sample and the corresponding sparsely annotated object set, respectively. For the i -th image, the annotated object set can be represented as O i = { x j i , y j i , b j i } N il j =1 , where x j i , y j i , b j i denote the image region, the class label, the bounding-box location of the j -th annotated object, and N il (i.e., N il ≥ 0 ) refers to the total number of labeled instances.

In the sparsely annotated aerial object detection task, how to mine more confident supervision (e.g., pseudo-labels) from these unlabeled images/regions is critical to promote the detector optimization. In general, most existing sparsely annotated object detection approaches (10; 26; 25) adopted a fixed threshold to select pseudo-labels, which can be used to provide more supervised signals. Unlike these typical objects, certain aerial objects characterized by their large, distinct features (e.g.,

planes) exhibit heightened predicted probabilities, whereas aerial objects with small dimensions (e.g., vehicles) tend to have lower predicted probabilities. Therefore, the imbalance in the predicted object probabilities could pose a challenge in pinpointing pseudo-label samples through a strict thresholding in the sparsely annotated aerial object detection problem. Inspired by the conformal prediction technique (20), which enables uncertainty quantification and utilizes confidence levels to tackle imbalanced data, we propose a novel Progressive Exploration-Conformable Learning (PECL) framework to address the SAOD task in aerial images.

3.2 PECL framework

Our proposed PECL framework can be built as a close-loop iterative learning process between the conformal pseudo-label exploration and the detector updating. Firstly, we can obtain a pretrained aerial detector Θ by employing these sparsely annotated objects in the training set. To provide the confident guide for the subsequent process, we construct class-wise prototypes P = { p ck ∈ R d } C,K c =1 ,k =1 with the fully connected layer features of each annotated instance as in (38). Here C is the number of classes, K is the number of prototypes per class, and d is the feature dimension of the prototype. We can obtain the candidate set of each input sample X i , i.e., { ˜ x t i } N ic t =1 , where N ic represents the number of candidates. We perform the conformal pseudo-label exploration to refine and identify these high-quality candidates as pseudo-labels, which can adaptively consider the distinct characteristics of different categories in the candidate set, especially in the aerial images. Here we develop the conformal pseudo-label explorer to facilitate the creation of an adaptive exploration policy, and the multi-clue selection evaluator to appraise the effectiveness of pseudo-label selection.

3.2.1 Conformal Pseudo-label Explorer

To address the imbalanced prediction probabilities across different categories in the aerial scenes, we specially design the conformal pseudo-label explorer to effectively probe these high-quality pseudolabels. Specifically, the conformal pseudo-label explorer π is a multi-layer perceptron composed of three fully connected layers. Its objective is to learn an adaptive pseudo-label exploration policy to determine whether to select the current candidate ˜ x t i . The conformal pseudo-label explorer π takes the current exploratory characteristic c t i as input, and obtains a two-dimensional selection probability distribution π ( c t i ) ∈ R 2 , representing the probabilities of taking different selections under the current characteristic. The selection decision a t i is then obtained by sampling from this distribution, i.e., a t i ∼ π ( c t i ) , where a decision value of 0 indicates not selecting the candidate, a value of 1 represents selecting the candidate as a pseudo-label. To assist the conformal pseudo-label explorer π in performing the rational decision, the important point is how to design a comprehensive exploratory characteristic c t i by considering multiple aspects of information. Here the characteristic c t i = F c { O i , ˜ O t i , ˜ x t i } is formed by utilizing the characteristics of the sparsely annotated objects O i , the selected pseudo-labels ˜ O t i , and the current candidate ˜ x t i in the current i -th aerial image. The exploratory characteristic thus provides a rich and accurate scene description for the conformal pseudo-label explorer, facilitating more effective pseudo-label selection.

Taking the candidate ˜ x t i as an example, we explain a detailed depiction of constructing the corresponding c t i at time t . The characteristic of the current candidate ˜ x t i can be represented as a feature vector, denoted as [ A P (˜ x t i ) , pro (˜ x t i ) , f sim (˜ x t i )] . A P (˜ x t i ) determines the precise level of confidence (20) for the candidate ˜ x t i to quantify the uncertainty of the predicted probability. Here A P (˜ x t i ) can be expressed as a class-conditional probability where the non-conformity scores of all learned prototypes take on a value bigger than or equal to the non-conformity score of this candidate under the condition of the certain class ˜ y t i . Formally:

where NC (˜ y t i | ˜ x t i ) = 1 -pro (˜ y t i | ˜ x t i ) represents the non-conformity score of the candidate ˜ x t i with the class ˜ y t i and I [ · ] is the indicator function. A higher non-conformity score indicates a greater dissimilarity between the candidate and the reference class distribution. pro (˜ x t i ) indicates the predicted probability of candidate ˜ x t i , inferred by the optimized detector. f sim (˜ x t i ) computes the

maximum cosine similarity between the feature of the candidate ˜ x t i and the prototypes { p ˜ y t i k } K k =1 of the predicted class label ˜ y t i . Based on the above, we can obtain current exploratory characteristic c t i by concatenating the average feature of the sparsely annotated objects O i and the selected pseudo-labels ˜ O t i with the feature of the current candidate ˜ x t i . The conformal exploratory characteristic enables the explorer to make effective pseudo-label selection and explore the unlabeled candidates efficiently, leading to the improved performance of the detector.

3.2.2 Multi-clue Selection Evaluator

To facilitate the conformal pseudo-label explorer π in learning optimal pseudo-label exploration policy, we introduce a multi-clue selection evaluator Q to assess the current exploratory characteristic c t i and selection policy a t i , providing instructive feedback for policy optimization. In essence, the selection evaluator is also constructed as a multi-layer network with three fully connected layers. It takes the current characteristic c t i and the decision policy a t i as input and estimates the cumulative reward for the characteristic-decision pairs (i.e., Q ( c t i , a t i ) ), signifying the effectiveness of the current policy. The cumulative reward takes into account the instant exploratory reward r t i accumulated at each time step t during the conformal pseudo-label decision, which measures how well the conformal pseudo-label explorer π performs. A high cumulative reward indicates that the conformal pseudo-label explorer π achieves positive candidate selection in decision-making process, and vice versa.

To accurately estimate the cumulative reward, the instant exploratory reward r t i should reflect whether the current selection a t i is appropriate or not. We first design a reward function ψ (˜ x t i ) to evaluate the impact of selecting current candidate ˜ x t i from the perspectives of information entropy and confidence margin, which is specifically represented as follows:

where ξ is the weighting parameter. ∆ H (˜ x t i ) represents the change in the information entropy of the current confident instances belonging to the same class ˜ y t i when the current candidate ˜ x t i is introduced into the collection of the sparsely annotated objects O i and the selected pseudo-labels ˜ O t i . It can be expressed as:

where H ( · ) calculates the information entropy of probability distribution. ¯ x i m is from the set { O i ∪ ˜ O t i ∪ P } ˜ y t i related to the class ˜ y t i and M denotes its total number. The candidate with a high uncertainty will increase the overall entropy. ∆ U (˜ x t i ) indicates the change in the confidence margin of the current confident instances belonging to the predicted class ˜ y t i when the current candidate ˜ x t i is selected. It can be formulated as:

where U ( · ) denotes the confidence margin, calculating the difference between the highest value and the second highest value in the prediction probability. A larger margin indicates that the candidate is more confident, leading to a positive promotion in the overall confidence margin.

By considering the reward function ψ (˜ x t i ) of the current candidate and the current selection a t i taken by the conformal pseudo-label explorer, we ultimately design the instant exploratory reward r t i as follows:

where the positive exploratory reward plays a constructive guiding role in the conformal pseudo-label exploration, and vice versa. Then the target value V t i can be defined as the sum of the instant exploratory reward r t i and the future cumulative reward Q ( c t +1 i , a t +1 i ) , formulated as follows:

Algorithm 1 The PECL training process

Input: Training set X = { X i , O i = { x j , y j , b j } N il } N

i i i j =1 i =1

Output: Detector Θ , conformal pseudo-label explorer π , and multi-clue selection evaluator Q

1: Pre-train Θ with X i , O i by minimizing L det + L pot , and construct the class-wise prototypes P ;

2: Initialize the experience replay pool D = ∅ ;

3: for i = 1 , 2 , 3 , ..., N do

4: Obtain the candidate set { ˜ x t i } N ic t =1 for X i with Θ ;

5: Initialize the selected pseudo-label set ˜ O 1 i = ∅ and the current candidate ˜ x 1 i , and then obtain the characteristic c 1 i ;

6: for t = 1 , 2 , 3 , ..., N ic do

7: # Perform the conformal pseudo-label explorer

8: Obtain the decision a t i using π with the current characteristic c t i ;

9: Update ˜ O t +1 i by taking the decision a t i ;

10: # Conduct the multi-clue selection evaluator

12: Calculate the reward r t i by Eq. (5);

11: Obtain the next characteristic c t +1 i with ˜ x t +1 i , ˜ O t +1 i and O i ;

13: Insert the recording ( c t i , a t i , r t i , c t +1 i ) into D ;

14: Sample a batch from D to update π and Q ;

15: end for

16:

Update Θ with annotated objects O i and the selected pseudo-labels ˜ O i by minimizing L det (Eq.(8)).

17: end for

where γ is a discount factor. Finally, we can continuously update the conformal pseudo-label explorer π and the multi-clue selection evaluator Q through performing gradient descent on the following objective function:

where the decision a t i is sampled from π ( c t i ) . The first term aims to adjust the predicted cumulative reward to approximate the target value for updating the multi-clue selection evaluator Q . The second term aspires to maximize the cumulative reward, enabling the conformal pseudo-label explorer π to refine the exploration policy. These selected high-quality candidates can in turn promote the capability of the aerial object detector. Furthermore, we use an experience replay pool D to store a series of data { ( c t i , a t i , r t i , c t +1 i ) } and sample batches to gradually update the parameters of the conformal pseudo-label explorer and the multi-clue selection evaluator (i.e., θ exp and θ eva ).

3.2.3 Progressive detector updating

The ultimate goal of our progressive exploration-conformable learning framework is to optimize an aerial detector by utilizing the sparse annotation and adaptively exploring these high-quality pseudo-labels in the training set. By employing these explored pseudo-labels ˜ O N ic i and the initial sparsely annotated objects O i as supervision signals, we can progressively acquire a satisfactory detector Θ by minimizing the loss function using stochastic gradient descent. Formally,

where | O i | and | ˜ O N ic i | represent the number of sparsely annotated objects and the selected pseudolabels during the conformal pseudo-label exploration process in the i -th image. ˜ y u i and ˜ b u i indicate the class label and bounding-box of the selected pseudo-labels ˜ x u i . L cls is the standard cross-entropy loss used in ReDet (6) and OR-CNN (29) or the focal loss used in S 2 A-Net (5) for classification, L reg is the SmoothL1 loss for localization. To construct representative class-wise prototypes for providing confident guidance, we adopt the prototype loss L pot as referenced in (38). To mitigate the impact of false negatives, we additionally employ the background weighting reduction strategy during the detector optimization process. Both the detector updating and the reinforced pseudo-label exploration are integrated to address the sparsely annotated aerial object detection task in a closed-loop, mutually reinforcing fashion. The PECL training process is presented in Algorithm 1.

4 Experiments

4.1 Dataset and setup

Datasets: We conduct experiments on two public aerial datasets, DOTA (27) and HRSC2016 (15), to evaluate the performance of our proposed PECL. The DOTA (27) dataset consists of 2806 highresolution aerial images, where the fully annotated 188282 instances vary in size, aspect ratio and rotation angle. The dataset has 15 object categories, including plane (PL), baseball-diamond (BD), bridge (BR), ground-track-field (GTF), small-vehicle (SV), large-vehicle (LV), ship (SH), soccer-ballfield (SBF), tennis-court (TC), basketball-court (BC), storage-tank (ST), roundabout (RA), harbor (HA), swimming-pool (SP), and helicopter (HC). To meet the requirement of sparse annotation, we adopt a novel protocol where we randomly sample 1%, 2%, 5% and 10% of instances for each class for annotation. While the HRSC2016 (15) dataset contains aerial images with complex backgrounds and diverse ship objects. The training set and validation set contain 436 and 181 images, while the test set consists of 444 images. Here 1% and 2% label rates often lead to over-fitting due to the HRSC2016 dataset is with a small amount of images, thus we only generate sparsely labeled versions of the dataset with 5% and 10% label rates. All evaluation experiments are conducted on the test set with the standard mean average precision (mAP) as the performance evaluation metric.

Implementation details: To demonstrate the generality of our PECL, we adopt two-stage detectors, including ReDet (6) and OR-CNN (29), as well as a one-stage detector, S 2 A-Net (5), serving as the baselines. In our experiments, we set the following hyper-parameters: K =10, ξ =1, γ =0.9. The feature dimension d is set to 1024 for ReDet and OR-CNN, 256 for S 2 A-Net. Our models are trained with the mmdetection (4)/mmrotate (39) framework on the DOTA dataset for 12 epochs, and on the HRSC2016 dataset for 36 epochs. All the experiments are conducted on two NVIDIA 2080Ti GPUs. For performing the detector optimization, we employ the SGD optimizer with an initial learning rate of 0.01. The learning rate is divided by 10 at the decay steps: 8 and 11 for DOTA, 24 and 33 for HRSC2016. The momentum and weight decay are set to 0.9 and 1e-4. The background/foreground weights are set to 0.3 and 1.0, respectively. The conformal pseudo-label explorer and multi-clue selection evaluator adopt the SGD optimization with learning rates of 1e-3 and 5e-4, respectively. The maximum capacity of an experience replay pool is set to 1000.

4.2 Experimental results

DOTA: We present the detection results for the oriented bounding box (OBB) task and horizontal bounding box (HBB) task on the DOTA (27) dataset, as shown in Table 1 and Table 2, respectively. In general, our PECL has demonstrated remarkable improvements over different baselines across all

label rates. For the OBB task, PECL achieves performance gains of 10.25%, 7.55%, and 15.62% at the 1% label rate compared with the baselines, indicating the effectiveness of our approach on different detectors. Furthermore, it is worth noting that our PECL shows greater performance improvement in some densely distributed small objects, such as small-vehicle (SV) and ship (SH), compared to individually presented large objects, such as baseball-diamond (BD) and soccer-ball-field (SBF) (33.05%, 46.05% vs 9.18%, 6.81% at the 1% label rate with ReDet (6)). This demonstrates that our PECL takes full advantage of the contextual relationships between instances.

Similarly, for the HBB task, our PECL boosts the performance of the detectors at all label rates, achieving at least 4.43% improvement. Importantly, as to the ReDet baseline, our sparsely trained model could achieve a competitive performance to the fully supervised one (69.95% vs 77.47%) by only using a label rate of 10%.

HRSC2016: We further conduct comparative experiments on the HRSC2016 dataset (15), and the performance comparisons of our PECL with different detector baselines for two tasks are presented in Table 3. Overall, our method has brought great improvements across various experimental settings. At the 10% label rate, our PECL achieves mAPs of 80.65%, 85.50% and 87.29% for the OBB task, surpassing the baselines with gains of 12.23%, 29.80% and 7.63%, respectively. Similar improvements are observed at the 5% label rate. These results underscore the effectiveness of our PECL even on relatively small-scale aerial dataset, further highlighting its efficacy in sparsely annotated aerial object detection task.

To demonstrate the superiority of our PECL, we further compare it with other state-of-the-art semi/sparse-supervised methods applied to the sparsely annotated aerial object detection task. To ensure fairness, all comparative experiments are conducted with ResNet50/ReR50-ReFPN on the DOTA dataset at the 5% label rate. The results are presented in Table 4. It can be observed that our PECL achieves the best performance of 67.06% with ReDet as the baseline, surpassing Unbiased Teacher, BRL, Co-mining, and Region-based by 2.32%, 2.02%, 1.71%, and 1.35%, respectively. When S 2 A-Net is used as the baseline, our PECL is 3.68% and 1.61% higher in mAP compared

55

60

base+BWR+OPC+PLS

base+BWR+OPC

base+BWR

base

65

70

45

50

55

60

65

70

1%

2%

5%

10%

base+IE+CM

base+CM

base+IE

base

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1e6

1.0

0.8

0.6

0.4

0.2

0.0

pseudo-label explorer

ASS=2

ASS=1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1e6

0.0

0.2

0.4

0.6

0.8

1.0

selection evaluator

ASS=2

ASS=1

to SOOD and Calibrated Teacher. The reason may be that our approach can explore high-quality pseudo-labels by addressing the imbalanced prediction probabilities across different categories in the aerial scenes, thereby mitigating the limitation of sparse annotation and boosting the detector optimization.

4.3 Ablation study

We conduct comprehensive ablation experiments to evaluate the performance of our framework under different settings. All the experiments are conducted with the ReDet baseline on the DOTA dataset at the 1% label rate for the OBB task, unless stated otherwise.

Effect of each component: As shown in Table 5, we give comparative results with three strategies when generating an optimal pseudo-label selection policy, including the conformal pseudo-label explorer, the multi-clue selection evaluator, and the experience replay mechanism. When only using the conformal pseudo-label explorer, we update the policy using the policy gradient method (22) and the detector achieves an mAP of 60.84%. It already outperforms the baseline, verifying the effectiveness of the conformal pseudo-label exploration mechanism. When further adopting the multi-clue selection evaluator, we can achieve more improvement with an mAP of 61.31%. The reason may be that the evaluator provides positive guidance for policy optimization. Moreover, the accumulated experience can alleviate overestimation and bring the best mAP of 63.72%.

Reward setting in the multi-clue selection evaluator: We study the effects of three factors of the designed instant exploratory reward, including the information entropy (IE), confidence margin (CM) and value binarization (VB) technique in Eq. (5). As presented in Figure 2, the detection results can be improved with different factors of the exploratory reward to some extent. These results indicate that combining information entropy and confidence margin as the exploratory reward is the most effective way to measure the uncertainty/quality of candidates and evaluate the pseudo-label selection. Besides, the VB technique ensures the stable convergence for policy optimization, further enhancing the detection performance.

Exploratory characteristic design in the conformal pseudo-label explorer: To validate the significance of different characteristics when forming the exploratory characteristic, Table 6 presents its comparative results with the predicted probability, feature similarity and confidence level. The predicted probability can measure the accuracy of the candidate, resulting in 1.32% performance boost. The feature similarity indicates the resemblance of the candidate to the class centers, leading to a gain of 2.55%. The confidence level reflects the inconsistency between the candidate and the corresponding prototypes, contributing to 2.82% improvement. These results confirm the rationality of our characteristic design and emphasize the importance of appropriate exploratory characteristic to enhance the performance.

Action space in the conformal pseudo-label explorer: We design two sizes of action space (ASS), namely the dimension of the probability vector obtained by the pseudo-label explorer. For 'ASS=1', the explorer outputs the probability of selecting the candidate, while 'ASS=2' gives the likelihood of the candidate being selected or not. Our approach achieves performance of 60.81% ('ASS=1') vs 63.72% ('ASS=2') on the DOTA dataset at the 1% label rate. Moreover, Figure 3 illustrates the loss curves of explorer and evaluator under different action spaces. We can observe when 'ASS=2', the loss values converge faster and reach lower convergence values compared to 'ASS=1'. The

reason may be that 'ASS=2' could provide a larger decision space, thereby promoting more flexible decision-making and improving the exploration capability of the pseudo-label explorer.

5 Conclusion and Discussion

In this paper, we propose a Progressive Exploration-Conformable Learning (PECL) framework to address the sparsely annotated aerial object detection problem, which can adaptively perform the conformal pseudo-label exploration from the large-scale unlabeled aerial images. By introducing the conformal pseudo-label explorer and the multi-clue selection evaluator, we formulate the pseudolabel exploration as a conformal decision-making problem. It can select these confident pseudolabels by considering the specific characteristics of different categories and inter-instance contextual relationships. Extensive experiments and ablation studies demonstrate the effectiveness of our PECL framework.

Limitations. This work also has some limitations. The first limitation is that our work is built for scenes with densely distributed objects, and its performance in general scenes may be suboptimal. Secondly, the application of reinforcement learning algorithms increases the overall training time. In the future, we will continue to explore the potential of sparsely annotated object detection and extend this PECL framework to other weakly supervised detection/segmentation tasks in general/aerial images to further reduce the annotation costs.

References

Walid Abdullah Al and Il Dong Yun. Partial policy-based reinforcement learning for anatomical landmark localization in 3d medical images. IEEE Transactions on Medical Imaging , 39(4):1245-1255, 2019.

David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems , 32, 2019.

Juan C Caicedo and Svetlana Lazebnik. Active object localization with deep reinforcement learning. In Proceedings of the IEEE International Conference on Computer Vision , pages 2488-2496, 2015.

Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 , 2019.

Jiaming Han, Jian Ding, Jie Li, and Gui-Song Xia. Align deep features for oriented object detection. IEEE Transactions on Geoscience and Remote Sensing , 60:1-11, 2021.

Jiaming Han, Jian Ding, Nan Xue, and Gui-Song Xia. Redet: A rotation-equivariant detector for aerial object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2786-2795, 2021.

Wei Hua, Dingkang Liang, Jingyu Li, Xiaolong Liu, Zhikang Zou, Xiaoqing Ye, and Xiang Bai. Sood: Towards semi-supervised oriented object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 15558-15567, 2023.

Chen Huang, Simon Lucey, and Deva Ramanan. Learning policies for adaptive tracking with deep feature cascades. In Proceedings of the IEEE International Conference on Computer Vision , pages 105-114, 2017.

Jisoo Jeong, Seungeui Lee, Jeesoo Kim, and Nojun Kwak. Consistency-based semi-supervised learning for object detection. Advances in Neural Information Processing Systems , 32, 2019.

Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Proceedings of International Conference on Machine Learning , volume 3, page 896, 2013.

Junnan Li, Caiming Xiong, and Steven CH Hoi. Comatch: Semi-supervised learning with contrastive graph regularization. In Proceedings of the IEEE International Conference on Computer Vision , pages 9475-9484, 2021.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision , pages 2980-2988, 2017.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of European Conference on Computer Vision , pages 740-755, 2014.

Yen-Cheng Liu, Chih-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. In Proceedings of International Conference on Learning Representations , 2021.

Zikun Liu, Liu Yuan, Lubin Weng, and Yiping Yang. A high resolution optical satellite image dataset for ship recognition and some new baselines. In International Conference on Pattern Recognition Applications and Methods , volume 2, pages 324-331, 2017.

Junwei Luo, Xue Yang, Yi Yu, Qingyun Li, Junchi Yan, and Yansheng Li. Pointobb: Learning oriented object detection via single point supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 16730-16740, 2024.

Romany F Mansour, José Escorcia-Gutierrez, Margarita Gamarra, Jair A Villanueva, and Nallig Leal. Intelligent video anomaly detection and classification using faster rcnn with deep reinforcement learning model. Image and Vision Computing , 112:104229, 2021.

Yusuke Niitani, Takuya Akiba, Tommi Kerola, Toru Ogawa, Shotaro Sano, and Shuji Suzuki. Sampling techniques for large-scale object detection from sparsely annotated objects. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6510-6518, 2019.

Sai Saketh Rambhatla, Saksham Suri, Rama Chellappa, and Abhinav Shrivastava. Sparsely annotated object detection: A region-based semi-supervised approach. In Proceedings of the IEEE International Conference on Computer Vision , 2023.

Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research , 9(3), 2008.

Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Lee, and Tomas Pfister. A simple semi-supervised learning framework for object detection. arXiv preprint arXiv:2005.04757 , 2020.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems , 12, 1999.

Yihe Tang, Weifeng Chen, Yijun Luo, and Yuting Zhang. Humble teachers teach better students for semi-supervised object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3132-3141, 2021.

Zhiqiang Tian, Xiangyu Si, Yaoyue Zheng, Zhang Chen, and Xiaojian Li. Multi-step medical image segmentation based on reinforcement learning. Journal of Ambient Intelligence and Humanized Computing , pages 1-12, 2020.

Haohan Wang, Liang Liu, Boshen Zhang, Jiangning Zhang, Wuhao Zhang, Zhenye Gan, Yabiao Wang, Chengjie Wang, and Haoqian Wang. Calibrated teacher for sparsely annotated object detection. In Proceedings of the AAAI Conference on Artificial Intelligence , 2023.

Tiancai Wang, Tong Yang, Jiale Cao, and Xiangyu Zhang. Co-mining: Self-supervised learning for sparsely annotated object detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 2800-2808, 2021.

Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in aerial images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3974-3983, 2018.

Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems , 33:6256-6268, 2020.

Xingxing Xie, Gong Cheng, Jiabao Wang, Xiwen Yao, and Junwei Han. Oriented r-cnn for object detection. In Proceedings of the IEEE International Conference on Computer Vision , pages 3520-3529, 2021.

Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-end semi-supervised object detection with soft teacher. In Proceedings of the IEEE International Conference on Computer Vision , pages 3060-3069, 2021.

Xue Yang, Gefan Zhang, Wentong Li, Xuehui Wang, Yue Zhou, and Junchi Yan. H2rbox: Horizontal box annotation is all you need for oriented object detection. arXiv preprint arXiv:2210.06742 , 2022.

Jihun Yoon, Seungbum Hong, and Min-Kook Choi. Semi-supervised object detection with sparsely annotated dataset. In Proceedings of the IEEE International Conference on Image Processing , pages 719-723, 2021.

Ke Yu, Chao Dong, Liang Lin, and Chen Change Loy. Crafting a toolchain for image restoration by deep reinforcement learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2443-2452, 2018.

Yi Yu, Xue Yang, Qingyun Li, Feipeng Da, Jifeng Dai, Yu Qiao, and Junchi Yan. Point2rbox: Combine knowledge from synthetic visual patterns for end-to-end oriented object detection with single point supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 16783-16793, 2024.

Han Zhang, Fangyi Chen, Zhiqiang Shen, Qiqi Hao, Chenchen Zhu, and Marios Savvides. Solving missing-annotation object detection with background recalibration loss. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing , pages 1888-1892, 2020.

Hongyu Zhou, Zheng Ge, Songtao Liu, Weixin Mao, Zeming Li, Haiyan Yu, and Jian Sun. Dense teacher: Dense pseudo-labels for semi-supervised object detection. In Proceedings of European Conference on Computer Vision , pages 35-50, 2022.

Kaiyang Zhou, Yu Qiao, and Tao Xiang. Deep reinforcement learning for unsupervised video summarization with diversity-representativeness reward. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmentation: A prototype view. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2582-2593, 2022.

Yue Zhou, Xue Yang, Gefan Zhang, Jiabao Wang, Yanyi Liu, Liping Hou, Xue Jiang, Xingzhao Liu, Junchi Yan, Chengqi Lyu, et al. Mmrotate: A rotated object detection benchmark using pytorch. In Proceedings of the 30th ACM International Conference on Multimedia , pages 7331-7334, 2022.

A Advancing together with big foundation models.

To adapt to the big model era, we adopt big-model CLIP for sparsely annotated object detection task. The comparisons are shown in Table 7. We observe that direct use of CLIP for pseudo-label inference cannot cause a large improvement, but additionally adding our PECL boosts the performance greatly. A possible reason is the difference between aerial images and general images (trained for CLIP).

B Comparison with SOTA oriented object detection methods.

We have compared with some SOTA oriented object detection methods (e.g., S 2 A-Net, OR-CNN and ReDet) in sparsely annotated task, as in Tables 1, 2, 3 of the submitted paper . Here, we add a comparison with the latest oriented method LSKNet-T on the DOTA dataset at the 5% label rate, as shown in Table 8. It indicates that our PECL can be well-generalized to SOTA oriented detectors.

Table 8: Performance comparison with SOTA oriented detector as baselines at the 5% label rate on the DOTA dataset.

C Performance comparisons with more label rates setting.

We follow the protocol of previous semi-supervised method (e.g., STAC) to utilize the extreme label rates of 1%, 2%, 5%, 10%, which actually consider the abundance of objects in aerial images. For the label rates, e.g., 30%, 50%, 70%, 100%, we also report results on the DOTA dataset in Table 9. It can be observed that our PECL can bring certain performance improvement under all sparse label rates, indicating the effectiveness of our method in the sparsely annotated aerial object detection tasks. However, under the 100% label rate, the performance deteriorates, possibly because the method introduces other label noise under full supervision, which affects the performance of the detector.

D Visualization results

The qualitative comparisons of detection results are visualized in Figure 4. We can observe that our proposed PECL outputs more meaningful and precise predictions than Region-based and the baseline ReDet on all different tasks and datasets, especially in some densely distributed small instances, i.e., plane/ship. But our method seems to be not precise enough in location, which is the future research. These visualization results demonstrate the superiority of our PECL in SAAOD task.

E Description of prototype loss

Here we describe in detail the prototype loss L pot in the Eq.(8) of the submitted paper . In our proposed PECL framework, we construct class-wise prototypes P to provide prior confident guidance

(a) ground truth

(c) Region-based

(d) Ours

DOTA

HBB

DOTA

OBB

HRSC2016

OBB

plane

large-vehicle

swimming-pool

basketball-court

ship

大

大

for the subsequent reinforced pseudo-label exploration process. In order to obtain more robust and representative prototypes, the prototype loss consists of two components, expressed as:

where L cont represents a contrastive learning loss between sparsely annotated instances and prototypes to distinguish prototypes of different categories, expressed as:

where cos θ ck = p ck · F ( x j i ) T is the cosine similarity between the feature vector of instance x j i and the c -th class k -th prototype, F ( · ) extracts the feature of the last fully connected layer. k j i = arg max k { cos θ y j i k } K k =1 represents the k j i -th prototype of class y j i . While L ret is a regularization term to minimize the distance between the instances and the intra-class prototypes. The term can be formulated as follows:

By applying these constraints, we constantly evolve the prototypes through momentum update method during the detector optimization. Formally,

where α ∈ [0 , 1] is a momentum coefficient.

F Ablation for prototype number

The number K determines the number of class centers and the size of the calibration set used for confidence level, greatly affecting the subsequent reinforced pseudo-label exploration process. We present the performance comparisons of different prototype numbers on DOTA dataset at the 1% label rate in Table 10. It could be observed that when K = 10, the detector performance reaches the best

63.72% mAP. When K =5, the performance decreases, but it is still a considerable performance. When K rises to 50, it brings a certain degree of drop. We believe that it may be due to the redundant class centers, or sparse annotation that cannot learn prototypes well. Therefore, K =10 can better represent the probability distribution of classes, calculate the confidence level of candidates, and construct better exploratory characteristic, bringing balance between representation and computational complexity.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We focus on the problem of sparsely annotated aerial object detection task, i.e., the Conformal Pseudo-label Explorer and Multi-clue Selection Evaluator are elaborated in Sec. .3.2.1 and Sec. .3.2.2. We provide extensive quantitative and qualitative experimental results in Sec. .4 and Appendix .F, .B, .C, .A. In addition, the visualization results are shown in Sec. .D.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [NA]

Justification: The paper has no limitation.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the implementation details in Sec. .4.1. In addition, the codes will be released after the paper is accepted.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: No codes are included in this submission, but the codes will be publicly available when paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental settings are described in Sec. .4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We conducted a large number of statistical experiments and published the most stable results. The results of some of the comparative experiments come from published work in the field. We have reproduced them and obtained consistent results, proving that our experimental results are real and credible, and that the experimental comparison is fair.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computer resources are described in Sec. .4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurlIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The open source code framework and dataset used in this study is introduced in Sec. .4.1.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary:\nThe paper addresses the challenge of sparsely annotated object detection (SAOD) in aerial images, a critical task for real-world aerial intelligence systems where annotations are limited. Acknowledging the difficulty posed by the imbalanced probabilities and confidences in predicted aerial objects, the paper proposes a novel Progressive Exploration-Conformal Learning (PECL) framework. This framework adaptively selects high-quality pseudo-labels to enhance detection performance. It comprises a conformal pseudo-label explorer and a multi-clue selection evaluator, which together form a decision-making paradigm for pseudo-label exploration. The paper also demonstrates that their method outperforms existing SOTA methods on the DOTA and HRSC2016 datasets.\nContributions:\nThe paper makes significant contributions to the field of semi-supervised aerial object detection by addressing key challenges related to sparse annotations and proposing a robust framework that improves detection performance through adaptive and progressive pseudo-label exploration.\n\nSTRENGTHS:\n(1) The overall paper is technically sound.\n(2) For originality and significance, the proposed PECL integrates a conformal pseudo-label explorer with a multi-clue selection evaluator to adaptively select high-quality pseudo-labels, offering a new perspective on handling sparse annotations in object detection.\n(3) For quality and clarity, the methodological framework is well-structured, detailing the iterative process between pseudo-label exploration and detector updating.\n\nWEAKNESSES:\n(1) While the PECL framework is innovative, its complexity might pose challenges for practical implementation. The multi-layer perceptron for the conformal pseudo-label explorer and the iterative training process could be computationally intensive. Discussing the impact of these modules on the overall computational cost and inference time is supposed to consider.\n(2) Can the authors provide more details on the computational efficiency of the PECL framework? Specifically, how does the iterative process impact training time, and are there any optimizations that can be applied to improve efficiency?\n(3) How does the PECL framework specifically address the detection of small and occluded objects in aerial images? Are there any additional strategies or modifications that could further enhance performance for these challenging cases?\n(4) The writing is a little unsatisfactory, especially in the first section, where the rationale for the proposed method is not well explained, making it difficult for readers to develop interest in your approach.\n(5) In Line 71-73, it is difficult for readers to intuitively realize that experimental performance is one of the main contributions without a very convincing data.\n(6) Shown in Line 142, “which can can adaptively” is a error. Moreover, the case of the first letter of the subsection and subsubsection should be unified in Section The Proposed Method.\n\nQUESTIONS:\nSee the weaknesses above.\n\nLIMITATIONS:\n(1) While the paper claims state-of-the-art performance, it lacks the comparative experiments with the SOTA methods from January to May 2024. Adding several comparative experiments with SOTA methods, such as on CVPR/TGRS 2024, can be considered.\n(2) For Section Related Work, it would be better to add some latest works published in 2024. For Section The Proposed Method, it would arouse the readers’ interest to add illustration chart of the proposed SCIR framework.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a Progressive Exploration-Conformal Learning (PECL) framework to address the sparsely annotated object detection task, which can adaptively perform the selection of high-quality pseudo-labels.  The pseudo-label exploration are formulated as a decision-making paradigm by adopting a conformal pseudo-label explorer and a multi-clue selection evaluator. Some evaluations on two public datasets demonstrate the superiority of our PECL\n\nSTRENGTHS:\n- Paper is well writen.\n- It seems interesting to simulate and learn the pseudo-label exploration process using reinforcement learning, i.e. the conformal pseudo-label explorer and the multi-clue selection evaluator.\n- The proposed method seems to work well.\n- The comparative experiments and ablation experiments are quite substantial.\n\nWEAKNESSES:\n- Table 4 lacks comparisons for weakly supervised oriented object detection, which are supported in MMRotate. Specifically, HBox-supervised (H2RBox, H2RBox-v2), Point-supervised (Point2RBox, PointOBB).\n- The authors mainly verified it on aerial image, but the proposed technology does not seem to be strongly related to the specific scenario, and therefore lacks verification in more convincing natural scenarios, e.g. COCO.\n- More object detection methods in aerial image need to be investigated in related work, such as OBB/HBB, full-/weakly-/semi-supervised.\n- After the introduction of reinforcement learning, the visualization of the changes of pseudo labels during the training process needs to be presented. The desired result is a significant improvement in quality.\n- The authors verified it on a CNN-based detector, and it would be more perfect if it could be verified on a DETR-based detector, e.g. ARS-DETR.\n- The left fields of Table 1 (e.g. 1%) and Table 4 (e.g. Sparse-annotated) do not seem to be centered. You can use ```\\multirow``` to adjust them.\n\nQUESTIONS:\n- I thought of an interesting setting, that is, Sparsely Annotated weakly-supervised roriented Object Detection, such as sparsely annotated HBox. Is it possible to add a related experiment based on H2RBox-v2 and verify it in combination with PECL techniques? Of course, this is not necessary, it's just my sudden thought. If the author can provide it, I am willing to further improve the rating.\n- I am not a professional reinforcement learning researcher, so I am worried whether the training of the entire detector will be slower after the introduction of reinforcement learning. I hope the authors can clarify this.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new learning framework, PECL, for sparsely annotated object detection (SAOD) in aerial images. The framework introduces a conformal pseudo-label explorer and a multi-clue selection evaluator, which can leverage category-specific characteristics and inter-instance contextual relationships. Comprehensive experiments demonstrate the superior performance of the proposed solution compared to prior approaches. Additionally, the paper includes numerous ablation studies and case studies that examine the various properties of the PECL framework.\n\nSTRENGTHS:\n- Aerial imagery has many useful properties for sparely annotated object detection. It is good that this paper explores those properties to develop a better SAOD framework. \n- The paper includes a substantial number of ablation studies and case studies, which could unveil valuable insights into the proposed method.\n\nWEAKNESSES:\n- The paper highlights PECL's effectiveness in addressing SAOD tasks by comparing the PECL-enabled version with a simple supervised baseline. The results are appealing, which is good. However, when comparing PECL with other SAOD solutions, the improvement is modest, at 1.35%.\n\n- The SAOD task in aerial imagery does not appear to be more challenging than SAOD tasks for general imagery, as objects in aerial imagery typically have only one degree of freedom in rotation. This reduces the significance of PECL's contribution.\n\n- The writing can be improved. The paper is somewhat hard to follow, for example, lines 128-131. Sometimes, even the logic is hard to follow. For instance, in line 31, the average number of objects per image seems irrelevant, as it is a dataset statistic. Neighboring aerial images could be stitched together to form a larger image with more objects.\n\nQUESTIONS:\nPlease check out the weaknesses section for details.\n\nLIMITATIONS:\nI don't see potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}