SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To accommodate the multiple segmentation sets within SAMRS, each having a different number of categories, we employ a multi-head pre-training strategy. This approach involves utilizing separate segmentation heads for individual datasets. The only distinction lies in the output channel count of the  $1 \times 1$  convolution, which corresponds to the number of categories. During batch-based training, diverse mini-batches are sampled from these sets to form a collective large batch, which is then fed into the network. Given the volume disparities among the various sets in SAMRS, proportional sampling is employed to obtain the mini-batches. Assuming a large batch of size  $B$  consists of  $M$  mini-batches with sizes  $B_{1}, B_{2}, \dots, B_{M}$ , it follows that  $B = B_{1} + B_{2} + \dots + B_{M}$ . Each mini-batch corresponds to its respective segmentation head, resulting in a training loss  $\mathcal{L}_i$  for the  $i$ th mini-batch. The total loss is computed as  $\mathcal{L} = \mathcal{L}_1 + \mathcal{L}_2 + \dots + \mathcal{L}_M$ . Assuming the sizes of the  $M$  sets in SAMRS are  $L_{1}, L_{2}, \dots, L_{M}$ , we can express this relationship as  $B_i = \frac{L_i}{\sum_{j=1}^{M} L_j} B$ ,  $i = 0, 1, \dots, M$ . Here,  $M$  can be easily extended for more RS detection datasets. In this study,  $M = 3$ . Figure 7 illustrates the pre-training pipeline. Each model is first pre-trained for 80k iterations with  $B = 96$  and then used for fine-tuning. All experiments are implemented by PyTorch on NVIDIA GeForce RTX 3090 GPUs.
4.2 Fine-tuning
4.2.1 Comparison to Various Pre-training Strategies
In the RS community, ISPRS Potsdam and IsAID are commonly-used finely annotated datasets for evaluating segmentation methods [25, 28, 36, 37, 44], and we use them to assess the pre-trained models, as Table 3-4 shown. It can be seen that, without good initialization, the performances of SEP are comparable as IMP but inferior to  $\mathrm{IMP + SEP}$ . On the Potsdam scene, for the traditional encoder-decoder network, SEP improves both convolutional and vision transformer networks, especially for UperNet and the backbones containing hierarchical features (also includes ResNet). As a result, ViTAEv2-S is greatly boosted and overperforms existing advanced methods in overall accuracy. We also observe that SEP is useful when combined with different pre-training strategies. Even if using the initialized weights generated by pre-training on SAMRS itself, SEP still can improve the accuracy,
Table 3: Segmentation results of different methods on the ISPRS Potsdam dataset.  $\ddagger$  : MAE pretraining on the MillionAID.  $\ddagger\ddagger$  : MAE pre-training on the SAMRS training set. “*” denotes the best score among all methods.
Method	Pretrain	Backbone	F1 score per category	OA	mF1
Imper. surf.	Building	Low veg.	Tree	Car
Comparison method
ST-UNet [15]	—	ResNet-50	79.19	86.63	67.89	66.37	79.77	—	86.13
ResUNet-a d7v2 [9]	—	—	93.50	97.20	88.20	89.20	96.40	91.50	92.90
LANet [11]	IMP	ResNet-50	93.05	97.19	87.30	88.04	94.19	90.84	91.95
DCFAM [39]	IMP	Swin-S	94.19	97.57	88.57	89.62	96.31	92.00	93.25*
Convolutional network
UNet	SEP	ResNet-50	90.62	94.75	85.12	83.91	96.51	89.70	90.18
UNet	IMP	ResNet-50	90.78	94.78	85.23	84.76	96.81	89.94	90.47
UNet	IMP+SEP	ResNet-50	91.36	94.92	85.39	85.24	97.17	90.29	90.82
UperNet	SEP	ResNet-50	91.02	94.82	84.28	83.97	96.95	89.70	90.21
UperNet	IMP	ResNet-50	90.70	94.44	84.68	83.94	96.58	89.59	90.07
UperNet	IMP+SEP	ResNet-50	91.38	95.26	85.14	84.88	97.16	90.27	90.76
Hierarchical vision transformer
UperNet	IMP	Swin-T	93.09	96.74	86.99	86.45	91.12	91.44	90.88
UperNet	IMP+SEP	Swin-T	93.06	96.65	87.07	86.74	97.64	91.88	92.23
UperNet	IMP	ViTAEv2-S	92.54	96.54	86.11	86.13	91.31	91.00	90.52
UperNet	IMP+SEP	ViTAEv2-S	93.45	96.99	87.65	87.00	97.67	92.25*	92.55
UperNet	IMP	InternImage-T	93.27	96.80	87.41	86.62	91.79	91.65	91.18
UperNet	IMP+SEP	InternImage-T	93.30	96.91	87.24	86.80	97.81	92.08	92.41
Plain vision transformer
UperNet	IMP	ViT-B	93.09	96.83	86.93	86.61	90.93	91.47	90.88
UperNet	IMP+SEP	ViT-B	92.96	96.52	86.62	86.01	97.57	91.60	91.94
UperNet	IMP	ViT-Adapter-B	93.16	96.77	87.09	86.71	91.20	91.53	90.98
UperNet	IMP+SEP	ViT-Adapter-B	93.20	96.75	87.06	86.52	97.68	91.91	92.24
Pre-training strategy
UNet	RSP	ResNet-50	91.49	95.42	85.70	85.18	97.05	90.49	90.97
UNet	RSP+SEP	ResNet-50	92.00	95.44	85.76	85.33	97.38	90.72	91.18
UperNet	RSP	ResNet-50	91.08	94.64	85.57	85.38	96.97	90.18	90.73
UperNet	RSP+SEP	ResNet-50	91.73	95.52	85.44	85.35	97.24	90.59	91.06
UperNet	BEiT	ViT-B	88.70	92.29	81.48	78.64	96.36	86.86	87.49
UperNet	BEiT+SEP	ViT-B	89.95	93.33	82.96	80.91	96.67	88.20	88.76
UperNet	MAE†	ViT-B + RVSA	92.67	96.38	86.43	85.89	90.46	90.97	90.37
UperNet	MAE+SEP	ViT-B + RVSA	92.69	96.33	86.28	85.60	97.56	91.33	91.69
UperNet	SAMRS-MAE‡‡	ViT-B + RVSA	92.46	96.10	86.18	85.59	90.35	90.71	90.13
UperNet	SAMRS-MAE+SEP	ViT-B + RVSA	92.34	95.88	86.06	85.32	97.54	91.01	91.43
End-to-end transformer
Mask2Former	IMP	ResNet-50	88.40	92.93	83.05	83.98	86.00	87.54	86.87
Mask2Former	IMP+SEP	ResNet-50	72.41	78.98	63.14	61.62	73.16	70.14	69.86
excluding the effect of data volume used for training. We notice SEP played a negative role in the end-to-end structure, it may be because the objects of SAMRS are too small, which is unfavorable for the region-based Mask2Former. In addition, the Mask2Former, which has obtained high accuracies on natural images, does not perform as well as UNet and UperNet on RSIs. These results indicate more refined parameter adjustments of Mask2Former are needed in later research. On the IsAID dataset, the performances of SEP on simple convolutional networks depending on local perception are unstable, because IsAID and DOTA share the same images but with different annotations, which may confuse the model. Benefiting from SEP, vision transformer networks are further enhanced and surpass previous methods. From these results we can see, SEP is able to mitigate the influence of task-level disparities, specifically the gaps between upstream pre-training tasks and downstream segmentation tasks.
4.3 Fine-tuning with Small-size Training Samples
The difficulty of annotating pixel-level masks limits the scale of existing remote sensing segmentation datasets, ultimately constraining the performance of trained models due to insufficient training samples. In order to investigate the effectiveness of SEP under conditions of limited training samples, we conducted experiments involving fine-tuning models using small fractions (1%, 3%, and 5%) of data from the ISPRS Potsdam and IsAID training sets, as outlined in Table 5 and Table 6. The integration of SEP with SAMRS, which provides a valuable segmentation prior, yields superior results compared to the IMP and RSP counterparts. This advantage is particularly evident when the number of available samples is scarce in the ISPRS Potsdam scene. Conversely, the results for the IsAID dataset exhibit an opposite trend due to the inherent challenges of this dataset, where both IMP and RSP yield extremely low overall accuracies. Nevertheless, the adoption of SEP significantly improves model performance. These findings highlight the importance of conducting segmentation
Table 4: Segmentation results of different methods on the IsAID dataset. ‡: MAE pre-training on the MillionAID. “*” denotes the best score among all methods.
Method	Pretrain	Backbone	IOU per category1	mIOU
SH	ST	BD	TC	BC	GTF	BR	LV	SV	HC	SP	RA	SBF	PL	HA
Comparison method
HMANet [28]	IMP	ResNet-50	65.38	70.92	74.71	88.69	60.51	54.57	28.98	59.74	50.28	32.58	51.41	62.88	70.20	83.79	51.91	62.64
UperNet	MAE ‡	ViTAE-B + RVSA	70.61	77.19	73.14	70.53	53.46	59.69	44.78	63.89	52.56	38.05	47.07	66.51	74.61	85.78	54.90	64.49
FactSeg [25]	IMP	ResNet-50	68.34	56.83	78.36	88.91	64.89	54.60	36.34	62.65	49.53	42.72	51.47	69.42	73.55	84.13	55.74	64.79
RssFormer [44]	IMP	RSS-B	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	65.88
Convolutional network
UNet	IMP	ResNet-50	50.11	48.50	31.52	74.30	15.38	14.99	8.83	51.59	34.71	0.07	39.03	7.88	39.33	73.12	47.71	35.80
UNet	IMP+SEP	ResNet-50	62.01	62.58	66.26	85.84	51.65	37.77	31.99	61.03	45.57	3.14	45.61	56.29	64.40	81.58	54.22	54.00
UNet	RSP	ResNet-50	53.00	53.53	62.99	78.25	34.48	36.92	24.83	54.95	34.22	2.11	40.36	42.18	50.37	77.82	48.81	46.32
UNet	RSP+SEP	ResNet-50	46.83	28.82	26.58	67.30	13.83	17.79	3.97	48.37	28.30	0.00	35.30	0.00	28.74	67.62	43.29	30.45
UperNet	IMP	ResNet-50	69.81	71.08	75.99	87.73	58.11	61.96	40.06	63.84	48.34	27.02	48.59	70.19	76.15	82.94	56.05	62.54
UperNet	IMP+SEP	ResNet-50	66.37	65.80	70.24	86.60	54.35	47.23	31.71	63.46	46.25	22.26	48.01	61.29	72.28	81.22	54.44	58.10
UperNet	RSP	ResNet-50	46.76	13.21	35.21	56.86	13.89	22.78	3.91	47.44	24.10	0.32	40.26	27.55	48.42	68.92	44.88	32.97
UperNet	RSP+SEP	ResNet-50	66.62	65.37	72.49	86.88	57.18	50.20	37.71	63.29	47.41	22.96	45.36	65.98	74.91	77.97	54.53	59.26
Vision transformer
UperNet	IMP	Swin-T	71.26	73.97	77.54	87.33	56.42	61.53	42.36	65.89	49.74	38.30	47.82	66.08	76.22	84.72	57.39	63.77
UperNet	IMP+SEP	Swin-T	73.37	74.19	76.63	87.63	56.40	61.44	42.42	65.98	50.92	38.82	49.16	69.70	75.05	84.83	57.27	64.25
UperNet	IMP	ViTAEv2-S	73.31	76.22	75.12	89.65	62.72	65.71	42.94	67.09	52.01	39.56	48.07	71.38	79.04	85.64	58.47	65.79
UperNet	IMP+SEP	ViTAEv2-S	75.95	75.48	75.38	89.56	63.79	64.85	45.62	68.49	53.27	37.56	49.49	69.84	79.22	85.85	59.58	66.26*
UperNet	IMP	InternImage-T	74.56	73.71	79.04	88.04	65.11	64.90	38.35	65.53	51.16	39.00	48.79	66.35	76.13	85.39	59.25	65.02
UperNet	IMP+SEP	InternImage-T	75.12	73.67	79.34	88.58	62.39	65.37	42.54	67.35	52.91	42.09	48.21	69.94	79.33	86.09	57.88	66.05
SH: ship. ST: storage tank. BD: baseball diamond. TC: tennis court. BC: baseball court. GTF: ground track field. BR: bridge. LV: large vehicle. SV: small vehicle. HC: helicopter. SP: swimming pool. RA: roundabout. SBF: soccer ball field. PL: plane. HA: harbor.
Table 5: Segmentation results of different pretraining methods on the ISPRS Potsdam dataset.
Method	Pretrain	Backbone	mFl
5%	3%	1%
UNet	IMP	ResNet-50	80.45	75.03	65.72
UNet	IMP+SEP	ResNet-50	81.80	77.97	69.70
Δ	+1.35	+2.94	+3.98
UNet	RSP	ResNet-50	80.34	75.35	62.86
UNet	RSP+SEP	ResNet-50	81.69	77.40	68.35
Δ	+1.35	+2.05	+5.49
Table 6: Segmentation results of different pretraining methods on the IsAID dataset.
Method	Pretrain	Backbone	mIOU
5%	3%	1%
UNet	IMP	ResNet-50	5.33	5.19	1.33
UNet	IMP+SEP	ResNet-50	18.74	12.98	7.04
Δ	+13.41	+7.79	+5.71
UNet	RSP	ResNet-50	4.68	3.67	2.50
UNet	RSP+SEP	ResNet-50	22.47	13.88	8.84
Δ	+17.79	+10.21	+6.34
pre-training using large-scale RS segmentation data, such as SAMRS, prior to training with limited data. Notably, the developed pipeline enables the rapid construction of such a dataset at a low labeling cost, making it a promising approach.
4.4 Limitations and Discussion
Previous classical datasets, such as HRSC2016 [22] and COCO [19], simultaneously contain bounding box and pixel-level mask annotations, proving the feasibility of the coexistence for segmentation and detection labels. Therefore, it is reasonable to construct the SAMRS dataset by transforming existing RS object detection datasets. However, despite successfully establishing the SAMRS dataset, which outperforms existing high-resolution RS segmentation datasets by more than tenfold, its volume remains smaller than large-scale classification datasets such as ImageNet [7] and MillionAID [23], commonly employed for pre-training purposes. Our current investigation focuses exclusively on pre-training small-scale basic models (about 100M) and we intend to incorporate larger models. Additionally, it is also worth trying to explore the impact of pre-training on SAMRS for tasks such as instance segmentation and object detection.
This study presents an effective way to create a large-scale remote sensing (RS) segmentation dataset by harnessing the capabilities of the Segment Anything Model (SAM) and existing object detection datasets. Given the unique challenges associated with RS data labeling, we investigate the performance of various prompts to identify the optimal settings for SAM. By leveraging these optimal settings, we generate extensive mask annotations for RS images, thereby creating a large-scale segmentation dataset named SAMRS. Remarkably, SAMRS surpasses all previously available high-resolution RS segmentation datasets in terms of volume. Furthermore, our statistical analysis reveals that SAMRS encompasses a diverse array of categories exhibiting varying sizes and distributions. SAMRS can be utilized for semantic segmentation, instance segmentation, and object detection, either independently or in combination. Specifically, we present a preliminary investigation and demonstrate the value of segmentation pre-training on SAMRS for RS segmentation tasks, especially in scenarios with limited training samples.
We acknowledge the authors of SAM for releasing codes and models, and the authors of DOTA, DIOR, and FAIR1M for providing their datasets. This work was supported in part by the National Natural Science Foundation of China under Grant 62225113 and in part by the National Key Research and Development Program of China under Grant 2022YFB3903405.
[40] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In CVPR, pages 14408-14419, 2023.
[41] Syed Waqas Zamir, Aditya Arora, Akshit Gupta, Salman Khan, Guolei Sun, Fahad Shahbaz Khan, Fan Zhu, Ling Shao, Gui-Song Xia, and Xiang Bai. isaid: A large-scale dataset for instance segmentation in aerial images. In CVPRW, pages 28-37, 2019.
[42] Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in aerial images. In CVPR, June 2018.
[43] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, pages 418-434, 2018.
[44] Rongtao Xu, Changwei Wang, Jiguang Zhang, Shibiao Xu, Weiliang Meng, and Xiaopeng Zhang. RSSformer: Foreground saliency enhancement for remote sensing land-cover segmentation. IEEE Transactions on Image Processing, 32:1052-1064, 2023.
[45] Jielu Zhang, Zhongliang Zhou, Gengchen Mai, Lan Mu, Mengxuan Hu, and Sheng Li. Text2seg: Remote sensing image semantic segmentation via text-guided visual foundation models. arXiv preprint arXiv:2304.10597, 2023.
[46] Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao. Vitaev2: Vision transformer advanced by exploring inductive bias for image recognition and beyond. International Journal of Computer Vision, 131(5):1141-1162, 2023.
[47] Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048, 2023.