SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Di Wang $^{1}$ ,* Jing Zhang $^{2\dagger}$ , Bo Du $^{1\dagger}$ , Minqiang Xu $^{3}$ , Lin Liu $^{3}$ , Dacheng Tao $^{2}$ , Liangpei Zhang $^{4\dagger}$
$^{1}$ School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China
$^{2}$ School of Computer Science, Faculty of Engineering, The University of Sydney, Australia
$^{3}$ National Engineering Research Center of Speech and Language Information Processing, China
$^{4}$ State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, China
{d_wang, dubbedo,zlp62}@whu.edu.cn; jing.zhang1@sydney.edu.au;
{mqxu7,linliu}@iflytek.com;dacheng.tao@gmail.com
The success of the Segment Anything Model (SAM) demonstrates the significance of data-centric machine learning. However, due to the difficulties and high costs associated with annotating Remote Sensing (RS) images, a large amount of valuable RS data remains unlabeled, particularly at the pixel level. In this study, we leverage SAM and existing RS object detection datasets to develop an efficient pipeline for generating a large-scale RS segmentation dataset, dubbed SAMRS. SAMRS totally possesses 105,090 images and 1,668,241 instances, surpassing existing high-resolution RS segmentation datasets in size by several orders of magnitude. It provides object category, location, and instance information that can be used for semantic segmentation, instance segmentation, and object detection, either individually or in combination. We also provide a comprehensive analysis of SAMRS from various aspects. Moreover, preliminary experiments highlight the importance of conducting segmentation pre-training with SAMRS to address task discrepancies and alleviate the limitations posed by limited training data during fine-tuning. The code and dataset will be available at SAMRS.
The advancement of earth observation technologies has led to the generation of abundant remote sensing images (RSI). These images retain valuable information about the spatial distribution and condition of extensive ground surfaces and geospatial objects, and can be conveniently accessed in real-time. Consequently, remote sensing data has garnered the interest of various disciplines, including agricultural monitoring, urban planning, and environmental protection. In particular, the identification of surface targets has been a fundamental task in these fields for several years.
To our knowledge, a significant number of RSIs remain unlabeled. Unlike natural images that can be easily comprehended by the human eye, interpreting RSI taken from an aerial perspective typically demands specialized expertise from practitioners. Furthermore, RSI objects are often distributed sparsely, and the images frequently contain small targets, making the labeling process less efficient. Therefore, the annotation of RSI has traditionally required substantial labor and time costs. Among
*This work was partially done during Di Wang's internship at iFlytek.
Figure 1: Some examples of SAM segmentation results on RSIs: (a) RGB aerial image obtained from the IsAID dataset [41]. (b) Airborne aerial image composed of near-infrared, red, and green bands. This image is from the ISPRS Vaihingen dataset<sup>1</sup>. (c) RGB satellite image observed by GF-2 sensors. This image is from the GID dataset [34]. (d) Hisea-1 SAR image from the Marine Farms Segmentation track of the 5th Gaofen Challenge<sup>2</sup>. These segmentation results are generated by the SAM demo website<sup>3</sup>.
various RS tasks, the classification task requires only a single category for the entire scene, and the detection task involves the additional step of bounding box annotation, while segmentation is particularly challenging since it necessitates pixel-level annotations to accurately delineate object boundaries.
Do we have to spend a significant amount of time annotating RSIs? The answer is probably no. Recently, the segment anything model (SAM) [17], which excels in object segmentation, has gained popularity as a new research focus in the field of computer vision. SAM accurately captures object locations and contours (i.e., in the form of masks), enabling it to distinguish various objects in the foreground and background. Furthermore, SAM possesses an impressive zero-shot segmentation ability, exhibiting high performance even when applied to specialized scenarios such as cell images photographed by microscopes [8] and medical images [26], despite being trained on a vast dataset of natural images. In the RS field, [31] firstly tests the performance of SAM on six public datasets. [16] extra introduce a domain decoder to improve the performance of SAM on the planetary geological mapping task. Beyond default prompts, [29, 45] consider utilizing texts as the prompt by adopting Grounding DINO [20] to obtain boxes that can be employed by SAM. Then, [29] realizes the one-shot segmentation with the help of PerSAM [47], while [45] applies the heatmap obtained from CLIP [30] to further optimize segmentation results. Different from the above methods with manual prompts, [3] design a prompter to adaptively generate prompts for improving the performance of SAM in instance segmentation. In addition, SAM is also used in producing rotated bounding boxes  $^4$ , which is significant for RS oriented object detection.
We have also found it performs well in recognizing diverse targets in RSI, even when the images are obtained using sensors that perceive different bands, such as infrared and microwave, or with varying resolutions, such as airborne or satellite imagery, as illustrated in Figure 1. Although we acknowledge that SAM may not have fully detected all regions, we believe that it has significant potential to improve the efficiency of annotating RSIs since it delivers promising segmentations on
1https://www.isprs.org/education/benchmarks/UrbanSemLab/2d-sem-label-vaihingen.aspx
$^{2}$ https://www.gaofen-challenge.com/challenge
<sup>3</sup>https://segment-anything.com/demo
4https://github.com/Li-Qingyun/sam-mmrotate
recognized areas. Therefore, in this study, we aim to utilize SAM to efficiently construct a large-scale RS segmentation dataset by obtaining pixel-level annotations for RSIs. Ground objects in RSI possess definite category properties, which are essential for real RS recognition tasks. However, the segmentation maps produced by SAM lack such information, rendering them unsuitable for labeling RSIs. To address this issue, we notice the annotations in existing RS object detection datasets, which include category and bounding box information. With the aid of SAM, we can leverage such detection annotations to obtain pixel-level semantic labels and efficiently construct large-scale segmentation datasets. The obtained dataset is called Segment Anything Model annotated Remote Sensing Segmentation dataset (SAMRS). SAMRS inherits the characteristics of existing RS object detection datasets that have more samples and categories compared with existing high-resolution RS segmentation datasets.
Since we efficiently obtain numerous segmentation label maps, it is natural to consider using the obtained dataset for pre-training. Existing models pretrained by classification tasks may be not very suitable for downstream tasks, e.g., segmentation, because of the task-level discrepancy [36], while the emergence of SAMRS is expected to address this issue. To this end, we train classical deep learning models on the SAMRS, and finetune the trained model on typical RS segmentation datasets to explore the feasibility of segmentation pre-training. The main contribution of this study can be summarized to: (1) We develop a SAM-based pipeline for efficiently generating RS segmentation annotations. (2) We obtain a large-scale RS segmentation dataset named SAMRS using existing RS object detection annotations, whose capacity is far beyond existing high-resolution RS segmentation datasets. (3) We conduct preliminary segmentation pre-training experiments on SAMRS. The results highlight the importance of conducting segmentation pre-training using large-scale RS segmentation data, such as SAMRS, for mitigating task discrepancy and dealing with limited training data. We hope this research could significantly enhance the annotation efficiency of RSIs, thereby unlocking the full potential of RS models, especially in the context of segmentation tasks.
2 Implementation
2.1 Segment Anything Model
To perform segmentation, additional prompts are needed to guide SAM to locate the object of interest, in addition to the input image. SAM supports various prompts, such as points, boxes, and masks, which can be input into the model either alone or in combination. It is important to note that when using point prompts, it is necessary to indicate whether the points are foregrounds or backgrounds. In this study, we use detection annotations from existing datasets to obtain all kinds of prompts since they contain both location and category information.
In this study, we employ SAM on four public RS object detection datasets, namely HRSC2016 [22], DOTA-V2.0 [10], DIOR [18], and FAIR1M-2.0 [33]. DOTA, DIOR, and FAIR1M are three large-scale datasets [33]. HRSC2016 is primarily designed for ship detection and comprises only one category. In comparison to the other three datasets, it has the
smallest data volume. Additionally, in the testing set, 124 images possess bounding box annotations and pixel-level labels simultaneously, making it highly suitable for evaluating the accuracy of SAM annotations. Therefore, we conduct an ablation study on the testing set consisting of the aforementioned 124 images to determine the optimal configuration for SAM. Following this, we generate segmentation labels for the remaining datasets. To obtain a segmentation dataset with more images or
(c)
Figure 2: The differences between segmentation labels and mask prompts. (a) Pixel-level annotated map from the original dataset. (b) Pixel-level annotations along with horizontal and rotated box ground truths. (c) Mask prompts derived from horizontal boxes. (d) Mask prompts derived from rotated boxes. The ship instances are marked with different colors by following (a).
categories, we opt for the latest versions of DOTA and FAIR1M. Based on the available annotations, we only transform the training and validation sets of DOTA-V2.0 and FAIR1M-2.0, while for DIOR, all data has been utilized. Here, according to the licenses, DOTA, DIOR, and FAIR1M can be used for academic purposes.
2.3 Prompt Settings
As RSIs are captured from an overhead perspective, the objects in them can have arbitrary orientations, unlike natural image objects that are typically oriented upward due to gravity. Hence, in addition to the usual horizontal bounding boxes (H-Box), we also consider oriented bounding boxes or rotated bounding boxes (R-Box) as box prompts. However, SAM does not directly support R-Box prompts. To address this issue, we use the minimum circumscribed horizontal rectangle of the R-Box, which is denoted as "RH-Box". It is also worth noting that the instances in the HRSC2016 testing set contain both H-Box and R-Box ground truth annotations.
In the case of the point prompt, due to the intricate shapes of various RS objects, such as airplanes, we have taken a cautious approach and only consider the center point as the foreground. We did not include background points in our study, as accurately defining them in an automated way can be challenging without additional contextual information. Regarding the mask prompt, we define the region enclosed by corresponding boxes as the mask prompt. Figure 2 illustrates the differences between the adopted mask prompts and ground truth segmentation labels. In SAM, the mask is a single-channel score matrix where positive values denote the active area where the target is located, whereas negative values represent irrelevant areas. In our experiments, we assign the values in these two types of areas as 1,000 and -1,000, respectively.
In summary, we have obtained six basic prompts, namely center point (CP), H-Box, RH-Box, and
their corresponding masks, i.e., H-Box-M, R-Box-M, and RH-Box-M, as illustrated in Figure 3.
(f)
Figure 3: The adopted basic prompts. (a) CP. (b) H-Box. (c) RH-Box. (d) H-Box-M. (e) R-Box-M. (f) RH-Box-M. The dashed line is used for the convenience of visualization.
2.4 Ablation Study
In addition to the above basic prompts, we also investigate various combinations of prompts in this study. To conduct a comprehensive analysis, we compute two types of mean intersection over union (mIOU) metrics:  $\mathrm{mIOU}_I$  and  $\mathrm{mIOU}_P$ , which measure the similarity between the predicted segmentation mask and the ground truth label. The former is the average value of the IoU calculated on a per-instance basis, while the latter measures the pixel-level accuracy. Given the  $i$ th instance with intersection set  $I_i$  and union set  $U_i$ , and the number of instances  $N$ , we have:
$$
\mathrm {m I O U} _ {I} = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {I _ {i}}{U _ {i}} \quad \mathrm {m I O U} _ {P} = \frac {\sum_ {i = 1} ^ {N} I _ {i}}{\sum_ {i = 1} ^ {N} U _ {i}}. \tag {1}
$$
Table 1 presents the evaluation results of utilizing different prompts. The point prompt delivers the worst performance and negatively affects the accuracy of any prompt combinations. This could be attributed to the insufficient amount of foreground points, which cannot guide the model effectively. The mask prompt performs better than the point prompt, but it still cannot generate high-quality segmentation annotations. The highest accuracy achieved by a mask prompt is approximately  $60\%$ , which is still much lower than the optimal prompts. Furthermore, the mask prompt has a negative impact on the performance of box prompts. When solely adopting the H-Box prompt, we obtain the highest accuracy compared to the point and mask prompts. For the case of utilizing R-Box annotations, the RH-Box prompt also achieves satisfactory performance. From this experiment, we conclude that: if an RS object detection dataset only has R-Box annotations, then the RH-Box prompt
Table 1: Results of using different prompts on the HRSC2016 testing set consisting of 124 images.
CP	H-Box	H-Box-M	R-Box-M	RH-Box	RH-Box-M	mIOU1	mIOUp
Point
✓	16.14	2.72
H-Box
✓	89.97	79.40
✓	40.54	36.71
✓	✓	86.67	77.35
✓	✓	74.21	62.25
✓	✓	24.54	5.41
✓	✓	✓	59.71	49.30
R-Box
✓	65.54	59.78
✓	✓	26.49	4.97
RH-Box
✓	88.85	76.42
✓	34.63	31.81
✓	✓	83.55	72.67
✓	✓	66.23	52.75
✓	✓	23.71	5.10
✓	✓	✓	49.24	39.03
Table 2: Comparisons of different high-resolution RS segmentation datasets.
Dataset	#Images	#Category	#Channels	Resolution (m)	Image size	Instance	Fine-grained
ISPRS Vaihingen1	33	6	IR,R,G	0.09	2,494 × 2,064
ISPRS Potsdam5	38	6	IR,RGB	0.05	6,000 × 6,000
Zurich Summer [35]	20	8	NIR,RGB	0.62	1,000 × 1,150
Zeebruges [27]	7	8	RGB	0.05	10,000 × 10,000
DeepGlobe Land Cover [6]	1,146	7	RGB	0.5	2,448 × 2,448
UAVid [24]	420	8	RGB	-	4,096 × 2,160 or 3,840 × 2,160
GID [34]	150	15	NIR,RGB	1 or 4	6,800 × 7,200
Landcover.ai [2]	41	3	RGB	0.25 or 0.5	9,000 × 9,500 or 4,200 × 4,700
IsAID [41]	2,806	15	RGB	-	800 × 800 × 4,000 × 13,000	✓
LoveDA [38]	5,987	7	RGB	0.3	1,024 × 1,024
SAMRS
SOTA	17,480	18	RGB	-	1,024 × 1,024	✓
SIOR	23,4631	20	RGB	-	800 × 800	✓
FAST	64,147	37	RGB	-	600 × 600	✓	✓
To avoid data snooping, only the 11725 images corresponding to the original DIOR trainval dataset are used in subsequent pre-trainings.
should be used; otherwise, the H-Box prompt should be adopted. This consideration is applied in our later dataset transformations.
2.5 Dataset Transformation
For the FAIR1M-2.0 dataset, since it only contains R-Box annotations, we use the corresponding RH-Box as the prompt. For DOTA-V2.0 and DIOR, we directly adopt the H-Box prompt. Prior to transformation, we follow the common practice to crop images in DOTA and FAIR1M datasets to  $1,024 \times 1,024$  and  $600 \times 600$ , respectively, while images in DIOR are maintained at the size of  $800 \times 800$ . The resulting datasets are named SOTA (i.e., DOTA  $\rightarrow$  SOTA), SIOR (i.e., DIOR  $\rightarrow$  SIOR), and FAST (i.e., Fine-grAined object recognition in high-Resolution remote sensing imagery  $\rightarrow$  Fine-grAined Segmentation for high-resolution remoTe sensing imagery), respectively. These datasets constitute a comprehensive and large-scale remote sensing segmentation database, i.e., SAMRS.
3 SAMRS
3.1 Basic Information
The obtained segmentation labels are stored in *.png files. Pixel values are aligned with the object classes of source object detection datasets. The areas that have not been covered by the generated masks will be in a pixel value of 255. We present the comparison of our SAMRS dataset with existing high-resolution RS segmentation datasets in Table 2 from different aspects. With the available high-resolution RSI object detection datasets, we can efficiently annotate 105,090 images containing 1,668,241 instances based on SAM and the identified prompt settings (Sec. 2.4), which is more than ten times the capacity of existing datasets. Additionally, SAMRS inherits the categories
<sup>5</sup>https://www.isprs.org/education/benchmarks/UrbanSemLab/2d-sem-label-potsdam.aspx
Figure 4: Some visual examples from the three subsets of our SAMRS dataset. For the definition of classes, please refer to the supplementary material.
of the original detection datasets, which makes them more diverse than other high-resolution RS segmentation collections. It is worth noting that RS object datasets usually have more diverse categories than RS segmentation datasets due to the difficulty of tagging pixels in RSIs, and thus our SAMRS reduces this gap.
Specifically, the resulting FAST dataset is a large-scale fine-grained RS segmentation dataset that targets diverse vehicles and grounds, while SOTA and SIOR are segmentation datasets containing common object categories. For this reason, we did not unify their categories. In addition to the massive pixel-level semantic mask annotations, SAMRS includes instance mask and bounding box annotations. This means that it can be used to perform semantic segmentation, instance segmentation, and object detection, either individually or in combination. This feature sets SAMRS apart from the IsAID dataset, which was independently annotated from scratch on DOTA-V1.0 [42] images.
3.2 Statistics and Analysis
Figure 5: Statistics of the number of pixels and instances per category in SAMRS. The histograms for the subsets SOTA, SIOR, and FAST are shown in the first, second, and third columns, respectively. The first row presents histograms on a per-pixel basis, while the second row presents histograms on a per-instance basis. A list of category abbreviations is provided in the supplementary material.
To gain a deeper understanding of the characteristics of the SAMRS dataset, we conduct a thorough analysis of their capacity per category, including pixel and instance numbers. The results are presented
Figure 6: Statistics of the mask sizes in the subsets of SAMRS. (a) SOTA. (b) SIOR. (c) FAST.
in Figure 5. In this analysis, we only count instances that have valid masks. The figure indicates that SIOR has more balanced categories compared to SOTA and FAST. In the instance-level statistics, we observe a large number of vehicle annotations, particularly on small ships and cars, as they are common in the real world and frequently appear in RSIs. This could also be the goal of initially developing these detection datasets. For instance, DOTA-V2.0 focuses on small targets, while FAIR1M mainly aims to accurately distinguish between different types of vehicles. Furthermore, it is observed that some categories have a high number of pixels but a low number of instances, which is likely due to their large size. For instance, the expressway-service-area in SIOR and the football-field in FAST demonstrate this pattern.
In addition, we investigate the distribution of mask sizes in SAMRS, as shown in Figure 6. The results indicate that, in general, there are more instances with smaller sizes in all subsets. However, some differences exist between the subsets. Specifically, FAST has more small objects than the other two sets. Nevertheless, SOTA appears to have a higher number of extremely small targets (i.e.,  $<100$  pixels), since its source dataset DOTA-V2.0 is designed for small object detection. On the other hand, SIOR has a more smooth distribution of mask sizes compared to SOTA and FAST.
In Figure 4, we visualize some segmentation annotations from the three subsets in our SAMRS dataset. As can be seen, SOTA exhibits a greater number of instances for tiny cars, whereas FAST provides a more fine-grained annotation of existing categories in SOTA such as car, ship, and plane. SIOR on the other hand, offers annotations for more diverse ground objects, such as dam. Hence, our SAMRS dataset encompasses a wide range of categories with varying sizes and distributions, thereby presenting a new challenge for RS semantic segmentation.
4.1.1 Data and Model Settings
To investigate the influence of segmentation pre-training (SEP) using SAMRS, we adopt multiple segmentation frameworks, including typical encoder-decoder networks and the recently emerged end-to-end structure. In encoder-decoder networks, we utilize classical UNet [32] and commonly-used UperNet [43]. Different from the original U-Net that has five blocks in the decoder part, to be compatible with the typical hierarchical pyramid backbone network that outputs four levels of features, we replace the last block to a single  $2 \times$  bilinear upsampling layer, which is followed by a segmentation head that contains a  $1 \times 1$  convolution, a  $2 \times$  bilinear upsampling, and a ReLU activation function. For UperNet, the segmentation head only employs a  $1 \times 1$  convolution. To comprehensively explore the SEP, in addition to traditional convolutional networks such as ResNet [14], diverse backbones are used, including hierarchical vision transformers: Swin [21], ViTAEv2 [46] and InternImage [40], and non-hierarchical networks, including ViT [12], ViT-Adapter [4] and ViT-RVSA [37]. For the end-to-end structure, we choose the recent Mask2Former [5]. The SAMRS is split into two parts, one for pre-training, and another for validation, see the supplementary material for more details. In the data preprocessing, we employed common data augmentation techniques,
Figure 7: The pipeline of segmentation pre-training on SAMRS. Different colors represent the data stream of various sets. The yellow parts will be used in fine-tuning.
including random scaling, random horizontal and vertical flipping, random rotation by 90 degrees, and altering pixel values through random color jitter and gamma transformation. Moreover, to ensure a fair comparison with prior studies [7, 36], we randomly cropped the input images to a size of  $224 \times 224$ .
4.1.2 Training Settings
Intuitively, a well-initialized backbone network generates discriminative features at the beginning of training, thereby facilitating the optimization process of the decoder component. Since the SEP is expected to mitigate the gaps between pre-training and downstream tasks. To this end, before the segmentation pre-training phase, the selected model's backbone network is initialized using pretrained weights. In our experiments, to fully evaluate the SEP, besides basic supervised pre-training on ImageNet (IMP) [7], we also utilize the RSP [36] on the MillionAID Dataset [23]. In addition, the unsupervised pre-training weights are also involved, including BEiT [1] and MAE [13]. Here, the MAE pre-training is conducted on the MillionAID [37], while the BEiT is pretrained on the ImageNet.