SAMRS: Scaling-up Remote Sensing Segmentation Dataset with Segment Anything Model
Authors: Di Wang
Jing Zhang
Bo Du
Minqiang Xu
Lin Liu
Dacheng Tao
Liangpei Zhang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Remote Sensing
Segmentation
Segment Anything Model
Pre-training

Abstract: The success of the Segment Anything Model (SAM) demonstrates the significance of data-centric machine learning. However, due to the difficulties and high costs associated with annotating Remote Sensing (RS) images, a large amount of valuable RS data remains unlabeled, particularly at the pixel level. In this study, we leverage SAM and existing RS object detection datasets to develop an efficient pipeline for generating a large-scale RS segmentation dataset, dubbed SAMRS. SAMRS totally possesses 105,090 images and 1,668,241 instances, surpassing existing high-resolution RS segmentation datasets in size by several orders of magnitude. It provides object category, location, and instance information that can be used for semantic segmentation, instance segmentation, and object detection, either individually or in combination. We also provide a comprehensive analysis of SAMRS from various aspects.  Moreover, preliminary experiments highlight the importance of conducting segmentation pre-training with SAMRS to address task discrepancies and alleviate the limitations posed by limited training data during fine-tuning. The code and dataset will be available at https://github.com/ViTAE-Transformer/SAMRS

SAMRS: Scaling-up Remote Sensing Segmentation Dataset with Segment Anything Model
Di Wang'; Jing Zhang Bo Du' Minqiang Xu Lin Liu Dacheng Tao', Liangpei Zhang 'School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China
School of Computer Science, Faculty of Engineering. The University of Sydney, Australia 3 National Engineering Research Center of Speech and Language Information Processing, China
4 State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, China
{d_ dd_wang,ddue dubo, caaibo,zpp2220 z1p62]2@huuuuu.m edu. cn; jing.zhang' thang1@sydnery @sydney. edu. com; dacheng tao@gmail com com
{mqxu7,
Abstract
The success of the Segment Anything Model (SAM) demonstrates the significance data-centric machine learning. However, due to the difficulties and high costs associated with annotating Remote Sensing (RS) images. large amount of valuable RS data remains unlabeled, particularly at the pixel level. In this study, we leverage SAM and existing RS object detection datasets to develop an efficient pipeline for generating a large-scale RS segmentation dataset, dubbed SAMRS. SAMRS totally possesses 105,090 images and 1,668,241 instances, surpassing existing high-resolution RS segmentation datasets in size by several orders of magnitude. It provides object category, location, and instance information that can be used for semantic segmentation, instance segmentation, and object detection, either individually or in combination. We also provide comprehensive analysis of SAMRS from various aspects. Moreover, preliminary experiments highlight the importance of conducting segmentation pre- pre-trainn training with SAMRS to address task discrepancies and alleviate the limitations posed by limited training data during fine-tuning. The code and dataset will be available at SAMRS.
Introduction
The advancement of earth observation technologies has led to the generation of abundant remote sensing images (RSI). These images retain valuable information about the spatial distribution and condition of extensive ground surfaces and geospatial objects, and can be conveniently accessed in real-time. Consequently, remote sensing data has garnered the interest of various disciplines, including agricultural monitoring, urban planning, and environmental protection. In particular, the identification of surface targets has been a fundamental task in these fields for several years.
To our knowledge, significant number of RSIs remain unlabeled. Unlike natural images that can be casily comprehended by the human eyc, interpreting RSI taken from an acrial perspective typically often distributed
demands specialized expertise from practitioners. Furthermore, RSI objects sparsely, and the images frequently contain small targets, making the labeling process less efficient. Therefore, the annotation of RSI has traditionally required substantial labor and time costs. Among
'This work was partially done during Di Wang's intersship at iFlytek. *Corresponding author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

(c)
Figure 1: Some examples of SAM segmentation results RSIs: (a) RGB aerial image obtained from the IsAID dataset [41]. (b) Airborne aerial image composed of near-infrared, red, and green bands. This image is from the ISPRS Vaihingen dataset!. (c) RGB satellite image observed by GF-2 sensors. This image from the GID dataset [34]. (d) Hisea-1 SAR image from the Marine Farms Segmentation track of the 5th Gaofen Challenge". These segmentation results are generated by the SAM demo website?.
various RS tasks, the classification task requires only single category for the entire scene, and the detection task involves the additional step of bounding box annotation, while segmentation is particularly challenging since it necessitates pixel-level annotations accurately delineate object boundaries.
Do have to spend significant amount of time annotating RSIs? The answer is probably Recently, the segment anything model (SAM) [17], which excels object segmentation, has gained popularity as new research focus in the field of computer vision. SAM accurately captures object locations and contours (i.e. in the form of masks), enabling it to distinguish various objects in the foreground and background. Furthermore, SAM possesses impressive zero-shot segmentation ability, exhibiting high performance even when applied to specialized scenarios such as cell images photographed by microscopes [8] and medical images [26], despite being trained vast dataset of natural images. In the RS field, [31] firstly tests the performance of SAM on six public datasets. [16] extra introduce domain decoder to improve the performance of SAM the planetary geological mapping task. Beyond default prompts, [29, 45] consider utilizing texts as the prompt by adopting Grounding DINO [20] obtain boxes that can be employed by SAM. Then, [29] realizes the one-shot segmentation with the help of PerSAM [47], while [45] applies the heatmap obtained from CLIP [30] further optimize segmentation results. Different from the above methods with manual prompts, [3] design prompter to adaptively generate prompts for improving the performance of SAM in instance segmentation. In addition, SAM is also used in producing rotated bounding boxes which is significant for RS oriented object detection.
We have also found it performs well in recognizing diverse targets in RSI, even when the images are obtained using sensors that perceive different bands, such as infrared and microwave, or with varying resolutions, such as airborne or satellite imagery, as illustrated in Figure 1. Although we acknowledge that SAM may not have fully detected all regions. we believe that it has significant potential to improve the efficiency of annotating RSIs since it delivers promising segmentations on
sem-label-- sem--abbe---aihiige...
aspx httpppp////ggggaaadddddddlagggeo
htttpe///ww..gofennccallleee www.ffen-challeggeee.. www.:/ngg....c.amaaaaaae chal/hhaaleegeeee
ptatpe://sagmeet-aannttt anything waaphnn....cmaaaa com/demo
"nttps://github
http:///githuu..oo/Lii.. com/Li- conn/ii-iinguunsaann-mom materottttmmmmmottttee

---

recognized areas. Therefore, in this study, we aim to utilize SAM to efficiently construct large-scale RS segmentation dataset by obtaining pixel-level annotations for RSIs. Ground objects in RSI possess definite category properties, which are essential for real RS recognition tasks. However, the segmentation maps produced by SAM lack such information, rendering them unsuitable for laheling RSIs. To address this issue, we notice the annotations in existing RS object detection datasets, which include category and bounding box information. With the aid of SAM, we can leverage such detection annotations to obtain pixel-level semantic labels and efficiently construct large-scale segmentation datasets. The obtained dataset called Segment Anything Model annotated Remote Sensing Segmentation dataset (SAMRS). SAMRS inherits the characteristics of existing RS object detection datasets that have more samples and categories compared with existing high-resolution RS segmentation datasets.
Since we efficiently obtain numerous segmentation label maps, it is natural to consider using the obtained dataset for pre-training. Existing models pretrained by classification tasks may be not very suitable for downstream tasks, e-g., segmentation, because of the task-level discrepancy [36], while the emergence of SAMRS is expected to address this issue. To this end, we train classical deep learning models on the SAMRS, and finetune the trained model on typical RS segmentation datasets to explore the feasibility of segmentation pre-training. The main contribution of this study can be summarized to: (1) We develop a SAM-based pipeline for efficiently generating RS segmentation annotations. (2) We obtain a large-scald RS segmentation dataset named SAMRS using existing RS object detection amnotations, whose capacity is far beyond existing nigh-resolution RS segmentation datasets. (3) We conduct preliminary segmentation pre- tre-training experiments SAMRS. The results highlight the importance of conducting segmentation pre-training using large-scale RS segmentation data, such as SAMRS, for mitigating task discrepancy and dealing with limited training data. We hope this research could significantly enhance the annotation efficiency of RSIs, thereby unlocking the full potential of RS models, especially in the context of segmentation tasks.
2 Implementation
2.1 Segment Anything Model
To perform segmentation, additional prompts are needed to guide SAM to locate the object of interest, in addition to the input image. SAM supports various prompts, such as points, boxes, and masks, which can be input into the model either alone or in combination. is important to note that when using point prompts, it is nec- essary to indicate whether the points are fore- grounds or backgrounds. In this study, we use detection annotations from existing datasets to obtain all kinds of prompts since they contain both location and category information.
(c)
(d)
2.2 Datasets
Figure 2: The differences between segmentation labels and mask prompts. (a) Pixel-level annotated map from the original dataset. (b) Pixel-level an- notations along with horizontal and rotated box ground truths. (c) Mask prompts derived from horizontal boxes. (d) Mask prompts derived from rotated boxes. The ship instances are marked with different colors by following (a).
In this study, we employ SAM on four
public RS object detection datasets, namely HRSC2016 [22], DOTA-V2.0 [10], DIOR [18).
and I [33]. DOTA, DIOR, and FAIRIM are three large-scale datasets [33].
HRSC2016 is primarily designed for ship de- tection and comprises only one category. In comparison to the other three datasets, has the smallest data volume. Additionally, in the testing set, 124 images possess bounding box annotations and pixel-level labels simultaneously, making it highly suitable for evaluating the accuracy of SAM annotations. Therefore, we conduct an ablation study on the testing set consisting of the aforemen- tioned 124 images to determine the optimal configuration for SAM. Following this, we generate segmentation labels for the remaining datasets. To obtain segmentation dataset with more images or
3

---

categories, we opt for the latest versions of DOTA and FAIRIM. Based on the available annotations, we only transform the training and validation sets of DOTA-V2.0 and FAIR IM-2.0, while for DIOR. all data has been utilized. Here, according to the licenses, DOTA, DIOR, and FAIR M can be used for academic purposes.
2.3 Prompt Settings
As RSIs are captured from an overhead perspective, the objects in them can have arbitrary orientations, unlike natural image objects that typically oriented upward due gravity. Hence, in addition to the usual horizontal bounding boxes (H-Box), we also consider oriented bounding boxes or rotated bounding boxes (R-Box) box prompts. However, SAM does not directly support R- R-Box prompts. To address this issue, we use the minimum circumscribed horizontal rectangle of the R-Box, which is denoted as "RH-Box". It is also worth noting that the instances in the HRSC2016 testing set contain both H-Box and R-Box ground truth annotations.
In the case of the point prompt, due to the in- tricate shapes of various RS objects, such as airplanes, have taken cautious approach and only consider the center point as the fore- ground. We did not include background points in our study, as accurately defining them in an automated way can be challenging without ad- ditional contextual information. Regarding the mask prompt, we define the region enclosed by corresponding boxes as the mask prompt. Figure 2 illustrates the differences between the adopted mask prompts and ground truth segmentation labels. In SAM, the mask a single-channel score matrix where positive values denote the active area where the target is located, whereas negative values represent irrelevant areas. In our experiments, we assign the values in these two types of areas as 1,000 and -1,000, respectively.
(a)
(b)
(c)
(d)
(1)
Figure 3: The adopted basic prompts. (a) CP. (b)
H-Box. (c) RH-Box. (d) H-Box-M. (e) R-Box-
M. (f) RH-Box-M. The dashed line used for the convenience of visualization.
In summary, we have obtained six basic prompts, namely center point (CP), H-Box, RH-Box, and their corresponding masks, i.e., H-Box-M, R-Box-M, and RH-Box-M, as illustrated in Figure 3.
2.4 Ablation Study
In addition to the above basic prompts, we also investigate various combinations of prompts in this study. To conduct a comprehensive analysis, we compute two types of mean intersection over union (mIOU) metrics: mIOU, and mIOUp, which measure the similarity between the predicted segmentation mask and the ground truth label. The former is the average value of the IoU calculated on a per- instance basis, while the latter measures the pixel-level accuracy. Given the ith instance with intersection set I; and union set U, and the number of instances N, we have:
1 N
N
ΣN ΣM U
mIOU,
mIOUp
U
Table I presents the evaluation results of utilizing different prompts. The point prompt delivers the worst performance and negatively affects the accuracy of any prompt combinations. This could be attributed to the insufficient amount of foreground points, which cannot guide the model effectively, The mask prompt performs better than the point prompt, but it still cannot generate high-quality segmentation annotations. The highest accuracy achieved by mask prompt is approximately 60%, which is still much lower than the optimal prompts. Furthermore, the mask prompt has a negative impact on the performance of box prompts. When solely adopting the H-Box prompt, we obtain the highest accuracy compared to the point and mask prompts. For the case of utilizing R-Box annotations, the RH-Box prompt also achieves satisfactory performance. From this experiment, we conclude that: if an RS object detection dataset only has R-Box annotations, then the RH-Box prompt

---

Table 1: Results of using different prompts on the HRSC2016 testing set consisting of 124 images.
CF
H-Box
H-Box-M
R-Box-M
RH-Box
RH-Box-M
mloU,
mlOUp
Point
16.14
2.72
H-Box
89.97
79.40
40.54
36.71
86.67
77.35
74.21
62.25
/
24.54
5.41
/
59.71
49.30
R-Box
65.54
59.78
26.49
4.97
RH-Box
88.85
76,42
34.63
31.81
83.55
72.67
66.23
52.75
23.71
5.10
49.24
39.03
Table 2: Comparisons of different high-resolution RS segmentation datasets.
Dataset
#lmages
#Category
#Channels
Resolution (m)
Image size
Instance
Fine-gmined
ISPRS Vashingen
33
IR.R.G
0.09
2,494 2,064
ISPRS Possdam
38
IR.GGB
0.05
6,900 x 6,000
Zurich Summer [35]
20
NIR.GGB
0.62
1,000 1,150
Zeebruges
RGB
10,000
DeepClobe Land Cover [6]
1,146
RGB
0.5
2,448 2,448
UAVid [24]
420
RGB
4,096
2,160 or 3,840 2,160
150
NIR_GGB
7,200
Landcoverai [2]
RGB
0.25 or 0.5
9,000
9,500 or 4,200 x 4,700
IAID [41]
2,806
15
RGB
800 800 ~u 4,000 13,000
/
LoveDA
5,987
RGB
1,024
SAMRS
SOTA
17,480
RGB
1,024 1,024
SIOR
23.463
RGB
800 800
FAST
64,147
RGB
To avoid data snooping. only the 11725 images correspouding to the original DIOR trainval datasct are used in subsequent peo-trainings.
should be used; otherwise, the H-Box prompt should be adopted. This consideration is applied in our later dataset transformations.
2.5 Dataset Transformation
For the FAIR1M-2.0 dataset, since it only contains R-Box annotations, we use the corresponding RH-Box as the prompt. For DOTA-V2.0 and DIOR, we directly adopt the H-Box prompt. Prior to transformation, we follow the common practice to crop images in DOTA and FAIR 1M datasets to 1,024 1,024 and 600 x 600, respectively, while images in DIOR are maintained at the size 800 x 800. The resulting datasets are named SOTA (i.e.. DOTA + SOTA), SIOR (i.e., DIOR - SIOR), and FAST (i.e., Fine-grAined object recognition high- nigh-Resslutton remote sensing imagery - Fine- grAined Segmentation for high-resolution remoTe sensing imagery), respectively. These datasets constitute comprehensive and large-scale remote sensing segmentation database, i.e., SAMRS.
3 SAMRS
3.1
Basic Information
The obtained segmentation labels are stored in *.png files. Pixel values are aligned with the object classes of source object detection datasets. The areas that have not been covered by the generated masks will be in pixel value of 255. We present the comparison of our SAMRS dataset with existing high-resolution RS segmentation datasets in Table 2 from different aspects. With the available high-resolution RSI object detection datasets, we can efficiently annotate 105, 0900000 images containing 1,668,24 2,21 instances based on SAM and the identified prompt settings (Sec. 2.4), which is more than ten times the capacity of existing datasets. Additionally, SAMRS inherits the categories
chttps://wwe http://wwww.......t.tt. org/ http//dducaaiioo/ennnnaaaee BaraaarSrkke/errrraeaaaaaaa /adpeadaaaaaaadaeeeee /d sem- label- phtellpotsdam...om aspx
ht
5

---

SOTA
SIOR
GTF
BR
DA
STT
ESA
BC
BF
WD
FAST
DCS
MB
FB
TC
IN
SC
VAA
CT
DT
OV
TRL
B7
QA
Figure 4: Some visual examples from the three subsets of our SAMRS dataset. For the definition of classes, please refer to the supplementary material.
of the original detection datasets, which makes them more diverse than other high-resolution RS segmentation collections. It is worth noting that RS object datasets usually have more diverse categories than RS segmentation datasets due the difficulty of tagging pixels RSIs, and thus our SAMRS reduces this gap.
Specifically, the resulting FAST dataset is large-scale fine-grained RS segmentation dataset that targets diverse vehicles and grounds, while SOTA and SIOR are segmentation datasets containing common object categories. For this reason, did unify their categories. In addition to the massive pixel-level semantic mask annotations, SAMRS includes instance mask and bounding box annotations. This means that can be used to perform semantic segmentation, instance segmentation, and object detection, either individually or in combination. This feature sets SAMRS apart from the IsAID dataset, which was independently annotated from scratch on DOTA-VIL [42] images.
3.2 Statistics and Analysis
685
(6)
WAVIO OREACTIOO SUOERAA STSTT
Figure Statistics of the number of pixels and instances per category in SAMRS. The histograms for the subsets SOTA. SIOR, and FAST are shown in the first, second, and third columns, respectively. The first row presents histograms on a per-pixel basis, while the second row presents histograms on per-instance basis. A list of category abbreviations is provided in the supplementary material.
To gain deeper understanding of the characteristies of the SAMRS dataset, we conduct thorough analysis of their capacity per category, including pixel and instance numbers. The results presented

---

(3)
(6)
(6)
18
Nurbbe betewco
Number of Instrroe
Number
Maxt 438
Madi: 638
Hedi: ebe
Figure 6: Statistics of the mask sizes the subsets of SAMRS. (a) SOTA. (b) STOR. (C) FAST.
in Figure 5. In this analysis, only count instances that have valid masks. The figure indicates that SIOR has more balanced categories compared to SOTA and FAST. In the instance-level statistics, observe large number of vehicle annotations, particularly on small ships and cars, they common in the real world and frequently appear in RSIs. This could also be the goal of initially developing these detection datasets. For instance. DOTA-V2.0 focuses on small targets. while IMIRIM mainly aims to accurately distinguish between different types of vehicles. Furthermore, it is observed that some categories have a high number of pixels but low number of instances, which is likely due to their large size. For instance, the sepresswwa--serviceeorr in SIOR and the football-fiele FAST demonstrate this pattern.
In addition, we investigate the distribution of mask sizes in SAMRS, as shown in Figure 6. The results indicate that, in general, there are more instances with smaller sizes in all subsets. However, some differences exist between the subsets. Specifically, FAST has more small objects than the other two sets. Nevertheless, SOTA appears to have higher number of extremely small targets (i.e., <100 pixels), since its source dataset DOTA-V2.00 is designed for small object detection. On the other hand, SIOR has more smooth distribution of mask sizes compared to SOTA and FAST.
3.3 Visualization
In Figure 4. we visualize some segmentation annotations from the three subsets our SAMRS dataset. As can be seen, SOTA exhibits a greater number of instances for tiny cars, whereas FAST provides a more tneeggranedd annotation of existing categories in SOTA such as car, ship, and plane. SIOR on the other hand, offers annotations for more diverse ground objects, such as dam. Hence, our SAMRS dataset encompasses wide range of categories with varying sizes and distributions, thereby presenting new challenge for RS semantic segmentation.
Experiment
4.1
Pre-training
4.1.1 Data and Model Settings
To investigate the influence of segmentation pre-training (SEP) using SAMRS, adopt multiple segmentation frameworks, including typical encoder-decoder networks and the recently emerged end-to-end structure. In encoder-decoder networks, utilize classical UNet [32] and commonly- used UperNet [43]. Different from the original U-Net that has five blocks the decoder part, to be compatible with the typical hierarchical pyramid backbone network that outputs four levels of features, we replace the last block to a single 2x bilinear upsampling layer, which is followed by segmentation head that contains a 1xl convolution, 2x bilinear upsampling, and ReLU activation function. For UperNet, the segmentation head only employs a x convolution. To comprehensively explore the SEP, addition to traditional convolutional networks such as ResNet [14], diverse backbones are used, including hierarchical vision transformers: Swin [21], ViTAEv2 [46] and InternImage [40], and non-hierarchical networks, including ViT [12], ViT-Adapter [4] and ViT-RVSA [37]. For the end-to-end structure, we choose the recent Mask2Former [5]. The SAMRS split into two parts, one for pre-training, and another for validation, see the supplementary material
for more details. In the data preprocessing, we employed common data augmentation techniques,
7

---

Seghwed
41
4
Backoone Network
Segmentation
Decoder
Buglned
FAST. 4
Seghead
L/
SAMRS
Px
Figure 7: The pipeline of segmentation pre-training on SAMRS. Different colors represent the data stream of various sets. The yellow parts will be used in fine-tuning.
including random scaling, random horizontal and vertical flipping, random rotation by 90 degrees, and altering pixel values through random color jitter and gamma transformation. Moreover, to ensure a fair comparison with prior studies [7, 36], we randomly cropped the input images to size of 224 224.
4.1.2 Training Settings
Intuitively, a well-initialized backbone network generates discriminative features at the beginning of training. thereby facilitating the optimization process of the decoder component. Since the SEP is expected to mitigate the gaps between pre-training and downstream tasks. To this end, before the segmentation pre-trat pre-training phase, the selected model's 's backbone network is initialized using pretrained weights. our experiments, to fully evaluate the SEP, besides basic supervised pre-tre pre-training on ImageNet (IMP) [7], also utilize the RSP [36] on the MillionAID Dataset [23]. In addition, the unsupervised pre-trane pre-training weights are also involved, including BEiT [1] and MAE [13]. Here, the MAE pre-training is conducted the MillionAID [37], while the BEiT is pretrained the
ImageNet.
To accommodate the multiple segmentation sets within SAMRS, each having different number of categories, we employ a multi-head pre-training strategy. This approach involves utilizing separate segmentation heads for individual datasets. The only distinction lies in the output channel count of the 1 x convolution, which corresponds to the number of categories. During batch- based training, diverse mini-batcher fed into the network. Given the volume disparities among the various sets in SAMRS, proportional sampling is employed to obtain the mini- mani-bacches. Assuming large batch of size B consists of M mini-batches with sizes B1, B2,." ... BM it follows that B Bi + B2 ....4 Bar Each mini-batch corresponds to its respective segmentation head, resulting in training loss L, for the ith mini-batch. The total loss is computed as = L1 + L2 + LM Assuming the sizes of the M sets in SAMRS L,
sampled from these sets to form collective large batch, which is then
are L1, ...... ... LMs we can express this relationship as B, ΣM B, i = 0, 1, M. Here, M
can be easily extended for more RS detection datasets. this study, AM = 3. Figure illustrates the pre-training pipeline. Each model is first pre-trained for 80k iterations with B 96 and then used for fine-tuning. All experiments are implemented by PyTorch on NVIDIA GeForce RTX 3090 GPUs.
4.2
Fine-tuning
4.2.1
Comparison to Various Pre-training Strategies
In the RS community, ISPRS Potsdam and IsAID are commonly-used finely annotated datasets for evaluating segmentation methods [25, 28, 36, 37, 44], and we use them assess the pre-trained models, Table 3-4 shown. It can be seen that, without good initialization, the performances of SEP are comparable as IMP but inferior IMP+SEP. On the Potsdam scene, for the traditional encoder- decoder network. SEP improves both convolutional and vision transformer networks, especially for UperNet and backbones containing hierarchical features (also includes ResNet). As a result, ViTAEy2-S is greatly boosted and overperforms existing advanced methods in overall accuracy. We also observe that SEP useful when combined with different pre-training strategies. Even if using the initialized weights generated by pre-training on SAMRS itself, SEP still can improve the accuracy,

---

Table 3: Segmentation results of different methods on the ISPRS Potsdam dataset. 1: MAE pre- training on the MillionAID. # MAE pre-training on the SAMRS training set. *** denotes the best score among all methods.
Fl per eategory Low veg.
Method
Pretrain
Backbone
Imper. surf.
Ruilding
Tree
Car
mF1
Comparison merhod
ST-UNet 115]
ResNet-50
79.19
86.63
67.89
66.37
79.77
86.13
ResUNct-a d7v2
93.50
97.20
88.20
89.20
96.40
91.50
92.90
LANet [1]]
IMP
ReaNet-50
93.05
97.19
87.30
88.04
94.19
90.84
91,95
DCFAM
IMP
Swin-S
94.19
97:57
88.57
89.62
96.31
92.00
93.25
Convolutional network
UNet
SEP
RexNet-50
94.75
96.51
UNet
IMP
ResNet-5U
90.78
94.78
85.23
84.76
96.81
89.94
9U.47
UNet
IMP+SEP
ResNct-50
91.36
94.92
85.39
85,24
97.17
90.82
UpetNet
SEP
ReaNet-50
91.02
94.82
84.28
83.97
96.95
89.70
90,21
UperNet
IMP
ResNct-50
90.70
94.44
83,94
96.58
89:59
90.07
UperNet
IMP+SEP
ResNet-50
91.38
95.26
85.
84.88
97.16
90.27
90.76
Hierarchical vision transformer
UpetNet
IMP
Swin-T
93.09
96.74
86.99
86.45
91.12
91.44
90.88
UperNet
IMP+SEP
Swin-T
93.06
96.65
87.07
86.74
97.66
92.23
UperNet
IMP
VITAEV2-S
92.54
96.54
86.11
86.13
91.31
91.00
90.52
UperNet
IMP+SEP
VITAEv2-S
93.45
96.99
87.65
87.00
97.67
92.25*
92.55
UperNet
IMP
Interlmaae-1 I
93.27
96.80
87.41
86.62
91.79
91.65
91.18
UperNet
IMP+SEP
IntemImage
93.30
96.91
87.24
97.81
Plain vision transformer
UperNee
IMP
VIT-B VTTB
93.09
96.83
86.93
86.61
90,93
91.47
90.88
UperNet
IMP+SFP
92.96
96.52
86.62
86.01
97.57
91.60
91.94
UperNot
IMP
ViI-Adapter-B
93.16
96.77
87.09
86.71
91.20
91.53
90.98
UperNet
IMP+SEP
ViT-Adapter-R
93.20
96.75
87.06
86.52
97.68
91.91
92.24
Pre-trainiug strategy
UNet
RSP
ReaNet-50
91.49
95.42
85.70
85.18
97.05
90.49
90,97
UNet
RSP+SEP
ResNet-50
92.00
95.44
85.76
85.33
97.38
90.72
91.18
UperNet
RSP
ResNet-50
91.08
94.64
85.57
85,38
96.97
90.18
90.73
UperNot
RSP+SEP
ResNet-50
91.73
95.52
85.35
97.24
90.59
91.06
UperNet
BETT
VTT-
88.70
92.29
81.48
78.64
96.36
86.86
87.49
UperNot
BEIT+SEP
VIT-B
89.95
93.33
82.96
80.91
96.67
88.20
88.76
UperNet
MAE
VTT-B RVSA
92.67
96.38
86.43
85.89
90.46
90.97
90,37
UperNot
MAE+SEP
VIT-B RVSA
92.69
96.33
86.
85.60
97.56
91.33
91.69
UperNet
SAMRS MAMRS-MAE-E #
VTT-B RVSA
92.46
96.10
86.
85,59
90.35
90.71
90,13
UperNot
SAMRS-MAB-SEP
VIT-B RVSA
92.34
95.88
86.06
85,32
97.54
91.01
91.43
End-to-end transformer
Mas&2romeer
IMP
ResNett
88.40
92.93
83.05
83.98
86.00
87.54
86.87
Mack2Former
IMP
ResNet-50
72.41
78.98
63.14
61.62
73.16
70.14
69.86
excluding the effect of data volume used for training. We notice SEP played a negative role in the end-to-end structure, it may be because the objects of SAMRS are too small, which is unfavorable for the region-based Mask2Former, In addition, the Mask2Former, which has obtained high accuracies on natural images, does not perform as well as UNet and UperNet on RSIs. These results indicate more refined parameter adjustments of Mask2Former are needed in later research. On the IsAID dataset, the performances of SEP on simple convolutional networks depending on local perception are unstable, because IsAID and DOTA share the same images but with different annotations, which may confuse the model. Benefiting from SEP, vision transformer networks are further enhanced and surpass previous methods. From these results we can see, SEP is able to mitigate the influence of task-level disparities, specifically the gaps between upstream pre-training tasks and downstream segmentation tasks.
4.3 Fine-tuning with Small-size Training Samples
The difficulty of annotating pixel-level masks limits the scale of existing remote sensing segmentation datasets, ultimately constraining the performance of trained models due to insufficient training samples. In order to investigate the effectiveness of SEP under conditions of limited training samples, we conducted experiments involving fine-tuning models using small fractions of data from the ISPRS Potsdam and IsAID training sets, as outlined in Table 5 and Table 6. The integration of SEP with SAMRS, which provides valuable segmentation prior, yields superior results compared to the IMP and RSP counterparts. This advantage is particularly evident when the number of available samples is scarce in the ISPRS Potsdam scene. Conversely, the results for IsAID dataset exhibit an opposite trend due to the inherent challenges of this dataset, where both IMP and RSP yield extremely low overall accuracies. Nevertheless, the adoption of SEP significantly improves model performance. These findings highlight the importance of conducting segmentation
3%, and 5%)

---

Table 4: Segmentation results of different methods on the IsAID dataset. 3: MAE pre-training on the MillionAID. "*p denotes the best score among all methods.
Bacsbooe
TOU pM CHEPIE
Comparbaan mochel
TMMANT [19]
IMP
63.55
70.92
14.71
15.00
80.1
51.57
25 95
59.70
50.28
12.58
5141
62.58
10.30
53.70
51.11
61.54
UpaNNe
MAE:
VITAE-B XVSA?
7061
77.19
73.64
71.33
53.46
59.69
44.2 75
63.89
52.55
38.05
47.67
66.51
74.61
85.78
$4.90
64.49
Fatteg (25)
IMP
RoaNei-50
68.34
56.85
76.36
68.91
$4.80
$4.60
36.34
$2.85
40.33
42.77
51.47
09.42
73.55
$4.15
55.24
64.59
Redioonnr [a4]
RSS-B
65.88
Connlational aetwork
V1.1
7:55
UNd
IMP-SEP
RodKz190
60.01
63.48
66.35
65.84
51.05
13.35
11.99
61.05
65.57
1.14
45.61
56.70
51.31
54,00
USa
KSP
R2
53300
55.55
02399
19.25
34.48
36.92
58.3
54.85
54.22
2.11
40.36
02.18
50.37
71.82
43.31
46.35
UNe
RSP SEP
46.83
28.50
26.55
07.30
13.85
17.29
3,97
46.37
26.30
0.00
35.30
0.00
28.75 04
67.60
43.29
30.45
0031
73390
07:21
38.11
0:335
40105
03:34
2770
70115
51.04
90115
0254
UpeiNer
Re
66.37
70:24
47:23
51.71
22.26
58.10
24100
08.35
UpoNss
RSP-SEP
RodNc-50
66.82
65.37
72.49
96.89
57.16
50.20
50.71
65.29
47.41
22.96
45.36
63.98
74.90
77.97
54.33
59.26
Vaion nauffonee
Upanne
71225
7337
77:31
67:33
36322
61.33
02.35
55.89
27.04
26.30
01332
18311
16.22
34:72
37.37
83377
Uperke
D4P-SFP
Suir-T
71.77
76.63
67.61
90:93
69 00
64:38
UpaiNer
7331
75:12
65:11
52.01
71.36
58.47
IMP-SEP
V
66.30
Upans
Bertinge
7X91
74254
18101
8511
6430
1710
15.15
35.10
63101
UpenNerr
IMP SEP
Intaaiingg T
75.12
73,07
79.34
89.58
62,39
65.37
42.44
67.35
52.91
42.09
45.21
09.94
79.33
55.09
57.88
66:85
SH: ship. ST; storage tank. BD: baseball diamond. TC: tennis court. BC: baschall court. GTP: ground track field. BR: bridge. L.V: large
vehicle, SV: small vehicle, HC: helicopter. SP: swimming pool. RA: roundabour. SBF: soccer ball field. PL: plane. HA: harbor.
Table 5: Segmentation results of different pre-
Table 6: Segmentation results of different pre- training methods on the IsAlD) dataset.
training methods on the ISPRS Potsdam dataset.
Method
Pretrain
Backbone
mFI
3%
1%
Method
Pretrain
Backbone
5%
mIOU
3%
1%
UNet
IMP
R5
80.45
75.03
65.72
UNet
IMP
RosNet-50
5.33
1.33
UNet
IMP+SEP
ResNet-50
81.80
77.97
69.70
UNel
IMP+SEP
ResNet-50
18.74
12.98
7.04
A
+1.35
+2.94
+3.98
+5.71
UNet UNet
RSP
ResNet-50
80,34
75.35
62.86
UNei
RSP
50
4.68
3.67
2.30
RSP+SEP
ResNet-50 50
81.69
77.40
68.35
UNet
RSP+SEP
ResNet-50
13.88
8.84
A
+1.35
+2.05
+5.49
A
+17.79
+10.21
+6.34
pre-training using large-scale RS segmentation data, such as SAMRS, prior to training with limited data. Notably, the developed pipeline enables the rapid construction of such dataset at low labeling cost, making it a promising approach.
4.4 Limitations and Discussion
Previous classical datasets, such as HRSC2016 [22] and COCO [19], simultaneously contain bounding box and pixel-level mask annotations, proving the feasibility of the coexistence for segmentation and detection labels. Therefore, it is reasonable to construct the SAMRS dataset by transforming existing RS object detection datasets. However, despite successfully establishing the SAMRS dataset, which outperforms existing high-resolution RS segmentation datasets by more than tenfold, its volume remains smaller than large-scale classification datasets such as ImageNet [7] and MillionAID [23], commonly employed for pre-training purposes. Our current investigation focuses exclusively on pre- pre-training scall-scale basic models (about 100M) and we intend to incorporate larger models. Additionally, it is also worth trying explore the impact of pre-training on SAMRS for tasks such as instance segmentation and object detection.
5
Conclusion
This study presents an effective way create a large-scale remote sensing (RS) segmentation dataset by harnessing the capabilities of the Segment Anything Model (SAM) and existing object detection datasets. Given the unique challenges associated with RS data labeling, we investigate the performance of various prompts to identify the optimal settings for SAM. By leveraging these optimal settings, we generate extensive mask annotations for RS images, thereby creating large-scale segmentation dataset named SAMRS. Remarkably, SAMRS surpasses all previously available high- resolution RS segmentation datasets in terms of volume. Furthermore, our statistical analysis reveals that SAMRS encompasses a diverse array of categories exhibiting varying sizes and distributions. SAMRS can be utilized for semantic segmentation, instance segmentation, and object detection, either independently or in combination. Specifically, we present a preliminary investigation and demonstrate the value of segmentation pre-training on SAMRS for RS segmentation tasks, especially in scenarios with limited training samples.
10

---

Acknowledgments
We acknowledge the authors of SAM for releasing codes and models, and the authors of DOTA, DIOR, and FAIRI I for providing their datasets. This work was supported in part by the National Natural Science Foundation of China under Grant 62225113 and in part by the National Key Research and Development Program of China under Grant 2022YFB3903405.
References
[1] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT pre-training of image transformers. In ICLR, 2022.
[2] Adrian Boguszewski, Dominik Batorski, Natalia Ziemba-Jankowska, Tomasz Dziedzic, and
Anna Zambrzycka. Landcover. ai: Dataset for automatic mapping of buildings, woodlands,
water and roads from acrial imagery. In CVPR, pages 1102-1110, 2021.
[3] Keyan Chen, Chenyang Liu, Hao Chen, Haotian Zhang. Wenyuan Li, Zhengxia Zou, and Zhenwei Shi. Rsprompter: Learning to prompt for remote sensing instance segmentation based on visual foundation model. arXiv preprint arXiv:2306. 2306..6669, 2023.
[4] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2023.
[5] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, pages 1290-
1299, 2022.
[6] like Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar. Deepglobe 2018: A challenge to parse the earth through satellite images. In CVPRW, pages 172-181, 2018.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale
hierarchical image database. In CVPR. pages 248-255, 2009.
[8] Ruining Deng, Can Cui, Quan Liu, Tianyuan Yao, Lucas W Remedios, Shunxing Bao, Bennett Landman, Lee E Wheless, Lori A Coburn, Keith T Wilson, et al. Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging. arXiv preprint w3Xiv:2304.04155. 2023.
[9] Foivos Diakogiannis, François Waldner, Peter Caccetta, and Chen Wu. Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammeetry and Remote Sensing. 162:94-114, 2020.
[10] Jian Ding. Nan Xue, Gui-Song Xia, Xiang Bai, Wen Yang, Michael Yang, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Object detection in aerial images: A sarge-scale benchmark and challenges. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2021.
[11] Lei Ding, Hao Tang, and Lorenzo Bruzzone. Lanet: Local attention embedding to improve the semantic segmentation of remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, (9(1) 1::6))4446355.. 2020.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. JCLR, 2021.
[13] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000-16009, June 2022.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pages 770-778, 2016.
[15] Xin He, Yong Zhou, Jiaqi Zhao, Di Zhang, Rui Yao, and Yong Xue. Swin transformer embedding unet for remote sensing image semantic segmentation. IEEE Transactions on Geoscience and Remote Sensing, 60:1-15, 2022.
[16] Sahib Julka and Michael Granitzer. Knowledge distillation with segment anything (sam) model for planetary geological mapping. arXiv preprint arXiv:2305.00 07586, 2023.
[17] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson,
Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-YenTe Lo, Piotr Dollar, and Ross Girshick. Segment anything. In ICCV, pages 4015-4026, October 2023.
[18] Ke Li, Gang Wan, Gong Cheng, Liqiu Meng, and Junwei Han. Object detection in optical remote sensing images: A survey and new benchmark. ISPRS journal of photogranmetry and remote sensing, 159:296-307, 2020.
11

---

[19] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV,
pages 740-755. 2014.
[20] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for
open-set object detection. arXiv preprint arXiv:2303. 00599, 2023.
[21] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining
Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In ICCV,
pages 10012-10022, 2021.
[22] Zikun Liu, Liu Yuan, Lubin Weng, and Yiping Yang. A high resolution optical satellite image dataset for ship recognition and some new baselines. In ICPRAM. pages 324-331, 2017. [23] Yang Long. Gui-Song Xia, Shengyang Li, Wen Yang. Michael Ying Yang, Xiao Xiang Zhu, Liangpei Zhang, and Deren Li. On creating benchmark dataset for aerial image interpretation: Reviews, guidances, and million-aid. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:4205-4233 2021.
[24] Ye Lyu, George Vosselman, Gui-Song Xia, Alper Yilmaz, and Michael Ying Yang. Uavid: A semantic segmentation dataset for uav imagery. ISPRS journal of photogrammerry and remote sensing, 165:108-119 2020.
[25] Ailong Ma, Junjue Wang, Yanfei Zhong, and Zhuo Zheng. Factseg: Foreground activation- driven small object semantic segmentation in large-scale remote sensing imagery. IEEE Trunsuctions on Geoscience and Remote Sensing, 60;1-16, 2022.
[26] Jun Ma and Bo Wang. Segment anything in medical images. arXiv preprint 2rXiv:230..13300
2023.
[27] Diego Marcos, Michele Volpi, Benjamin Kellenberger, and Devis Tuia. Land cover mapping at very high resolution with rotation equivariant cnns: Towards small yet accurate models. ISPRS journal of photogrummeirry and remote sensing, 145:96-107, 2018.
[28] Ruigang Niu, Xian Sun, Yu Tian, Wenhui Diao, Kaiqiang Chen, and Kun Fu. Hybrid multiple attention network for semantic segmentation in aerial images. IEEE Transactions Geoscience and Remote Sensing, 60:1-18, 2022.
[29] Lucas Prado Osco, Qiusheng Wu, Eduardo Lopes de Lemos, Wesley Nunes Gonçalves, Ana Paula Marques Ramos, Jonathan Li, and José Marcato Junior. The segment anything model (sam) for remote sensing applications: From zero to one shot. arXiv preprint arXiv:2306. /6623, 2023.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CML, pages 8748-8763. PMLR, 2021.
[31] Simiao Ren, Francesco Luzi, Saad Lahrichi, Kaleb Kassaw, Leslie M Collins, Kyle Bradbury, and Jordan M Malof. Segment anything, from space? arXiv preprint arXiv:2304. J 13000, 2023.
[32] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241, 2015.
[33] Xian Sun, Peijin Wang. Zhiyuan Yan, Feng Xu, Ruiping Wang. Wenhui Diao, Jin Chen, Jihao
Li. Yingchao Feng, Tao Xu, Martin Weinmann, Stefan Hinz, Cheng Wang, and Kun Fu. Fairlm: benchmark dataset for fine- ane-grained object recognition in higo-resolution remote sensing imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 184:1 (6-130, 2022.
[34] Xin-Yi Yi Tong, Gui-Song Xia, Qikai Lu, Huanfeng Shen, Shengyang Li, Shucheng You, and Liangpei Zhang. Land-cover classification with high-resolution remote sensing images using transferable deep models. Remote Sensing of Environment, 237:1 1 11322, 2020.
[35] Michele Volpi and Vittorio Ferrari. Semantic segmentation of urban scenes by learning local
class interactions. In CVPRW, pages 2015.
[36] Di Wang, Jing Zhang, Bo Du, Gui-Song Xia, and Dacheng Tao. An empirical study of remote sensing pretraining. IEEE Transactions on Geoscience and Remote Sensing, 61:1-20, 2023. [37] Di Wang, Qiming Zhang, Yufei Xu, Jing Zhang, Bo Du, Dacheng Tao, and Liangpei Zhang. Advancing plain vision transformer toward remote sensing foundation model. IEEE Transactions on Geoscience and Remote Sensing, 61:1-15, 2023.
[38] Junjue Wang, Zhuo Zheng, Ailong Ma. Xiaoyan Lu, and Yanfei Zhong. Loveda: A remote sensing land-cover datasct for domain adaptive semantic segmentation. In NeurIPS Track on Datasets and Benchmarks, volume 1, 2021.
[39] Libo Wang, Rui Li. Chenxi Duan, Ce Zhang, Xiaoliang Meng, and Shenghui Fang. A novel transformer based semantic segmentation scheme for fine-resolution remote sensing images. IEEE Geoscience and Remote Sensing Letters, 19:1-5, 2022.
12

---

[40] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu. Hongsheng Li, al. Interniaage: Exploring large-scale vision foundation models with deformable convolutions. In CVPR, pages 14408-14419 2023.
[41] Syed Waqas Zamir, Aditya Arora, Akshita Gupta, Salman Khan, Guolei Sun, Fahad Shah- baz Khan, Fan Zhu, Ling Shao, Gui-Song Xia, and Xiang Bai. isaid: A large-scale dataset for instance segmentation in aerial images. In CVPRW, pages 28-37, 2019.
[42] Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie. Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in aerial images. In CVPR, June 2018.
[43] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, pages 418-434, 2018.
[44] Rongtao Xu, Changwei Wang, Jiguang Zhang, Shibiao Xu, Weiliang Meng, and Xiaopeng Zhang. Rssformer: Foreground saliency enhancement for remote sensing land-cover segmenta- tion. IEEE Transactions on Image Processing. 32:1052-1064, 2023.
[45] Jielu Zhang, Zhongliang Zhou, Gengchen Mai, Lan Mu, Mengxuan Hu, and Sheng Li. Text2seg: Remote sensing image semantic segmentation via text-guided visual foundation models. arXiv preprins ar3iv:2304 10597, 2023.
[46] Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao. Vitaev2: Vision transformer ad- vanced by exploring inductive bias for image recognition and beyond. International Journal of Computer Vision.
1411::111111122 2023.
[47] Renrui Zhang, Zhengkai Jiang. Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot. arXiv preprint 2055v:230..03048, 2023.
13


Reviews:
[{"review_id": "XtYIAkWPUe", "reviewer": "Reviewer_ETXB", "summary": "", "questions": "", "limitations": "The authors well discuss the limitations in this work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The analyses of various adopted basic prompts in Fig.3 is interesting.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "lC9nOauRto", "reviewer": "Reviewer_MpE3", "summary": "", "questions": "", "limitations": "The experiment's baseline, ResNet50 + UNet, is relatively simple. To further demonstrate the effectiveness of the proposed pre-training paradigm, the authors should consider using a state-of-the-art model, like Mask2former.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Provides a new perspective that SAM also has a great zero-shot segmentation ability.\n2. Authors sufficiently compare different SAM prompt settings according to performance on a certain test set.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Kglugsd1RU", "reviewer": "Reviewer_rfwi", "summary": "", "questions": "", "limitations": "The authors include a brief limitations section, but I think it adequately addresses the limitations and potential negative societal impact of their work, which I believe are minimal.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The authors carefully design methods to automatically \"translate\" the existing dataset labels to detailed segmentation labels by developing careful prompts for SAM. They describe the methodology in detail with clarity and quantitatively assess different labeling methods.\n- They provide useful quantitative analysis of the resulting labeled dataset to better understand the automatically labeled segmentation masks included in SAMRS.\n- They conduct several interesting pre-training experiments, comparing to image classification pre-training and evaluating in fine-tuning settings with a small amount of data, which help shed light on the utility of SAMRS.\n- Overall the paper is well-written, and both the methodology and described dataset are likely of interest to the broader research community. The methodology is general enough to be applied to other datasets, potentially beyond remote sensing, and the dataset will be a valuable resource to both the remote sensing and ML communities for improving ML methods on remote sensing data.\n- Ethical and social implications are minimal as the authors have mostly just (substantially) improved existing datasets.\n- The authors also provide a datasheet which includes a lot of useful information for users of the dataset.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "jBve8DMMDc", "reviewer": "Reviewer_55Yp", "summary": "", "questions": "", "limitations": "- Authors state that SAM is not working well on a range of classes in RSI, but just selected ones. This is in line with other literature in the field. However, when utilizing SAM to create data, this can result in datasets that are focusing on classes / images that already work well with existing approaches like SAM, making it difficult to identify blind spots of existing architectures in order to push SOTA in RS. I.e., is this dataset biased towards classes that are already working well with approaches like SAM?\n\n- Inconstancies, errors etc of other datasets used to create this dataset may translate into this new dataset.\n\n- It is not a really novel finding that performance improves with more available data in pretraining, as more data allows to train bigger models with more parameters. See all work going on on self-supervised, transformer-based MAE in RS. Please clarify on the final purpose of this dataset. For pretraining, we see very promising results based on self-supervised MAE. The work should compare to this (e.g.  Wang, D., Zhang, Q., Xu, Y., Zhang, J., Du, B., Tao, D., & Zhang, L. (2022). Advancing plain vision transformer toward remote sensing foundation model. IEEE Transactions on Geoscience and Remote Sensing, 61, 1-15.) and more broadly discuss alternative approaches to supervised pretraining.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The approach automates a lot of manual work and translates existing RS datasets for e.g. object detection to segmentation, which enlarges this dataset category. It makes use of very recent approaches in computer vision (i.e. SAM) to benefit the remote sensing field.\n\nOverall a nice approach to accelerate labeling, which is of great interest in the field of RS for many specific use cases (but less for pretraining!) at the moment - rendering this an important topic.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper underwent a thorough review process by four reviewers. The reviewers were divided, with half recommending rejection and the remaining reviewers expressing support for acceptance. A primary concern raised by Reviewer ETXB is the suitability of SAMRS for evaluation purposes. After a thorough examination of the submission, it is evident that the primary intent behind SAMRS is to provide a large-scale dataset for pre-training, with less emphasis on positioning it as a benchmark dataset. The authors have not strongly highlighted SAMRS as a benchmark in their submission. I believe that SAMRS holds significant value as a resource for large-scale pre-training in the field RS images. Therefore, I recommend accepting this paper, provided that the authors include necessary discussions and incorporate new experiments into the final version.
Decision: Accept (Poster)