Exploring Segment Anything Foundation Models for Out of Domain Crevasse Drone Image Segmentation

Authors: steven  wallace
aiden  durrant
william d harcourt
richard  hann
georgios  leontidis
  this
Venue: NLDL_org_2025_Conference
Published: 2024-09-05 19:00:46

Abstract: In this paper, we explore the application of Segment Anything (SAM) foundation models for segmenting crevasses in Uncrewed Aerial Vehicle (UAV) images of glaciers. We evaluate the performance of the SAM and SAM 2 models on ten high-resolution UAV images from Svalbard, Norway. Each SAM model has been evaluated in inference mode without additional fine-tuning. Using both automated and manual prompting methods, we compare the segmentation quantitatively using Dice Score Coefficient (DSC) and Intersection over Union (IoU) metrics. Results show that the SAM 2 Hiera-L model outperforms other variants, achieving average DSC and IoU scores of 0.43 and 0.28 respectively with automated prompting. However, the overall off-the-shelf performance suggests that further improvements are still required to enable glaciologists to examine crevasse patterns and associated physical processes (e.g. iceberg calving), indicating the need for further fine-tuning to address domain shift challenges. Our results highlight the potential of segmentation foundation models for specialised remote sensing applications while also identifying limitations in applying them to high-resolution UAV images, as well as ways to enhance further model performance on out-of-domain glacier imagery, such as few-shot and weakly supervised learning techniques.

Main paper: 
 introduction: 
 In recent years, deep learning foundation models that contain the transformer neural network architecture have gained significant traction in natural language processing and computer vision [1, 2] . Notably, large companies like Meta Facebook AI Research (FAIR) and Google are contributing to the evolution of deep learning research by publishing more pre-trained models open source [3] [4] [5] . Convolutional Neural Networks (CNNs) and the Vision Transformer (ViT) have advanced the field by providing researchers with access to many pre-trained models that would otherwise be computationally prohibitive to pre-train from scratch on non-commercial computer hardware [6, 7] . In most cases, pre-training on a larger dataset in a controlled laboratory with more compute power and then fine-tuning to a specialist downstream task usually leads to better performance at a much cheaper cost [8] . However, foundation models still face challenges in generalising to specialised image segmentation applications from the Earth Observation (EO) and medical image domains because of the large quantities of labelled data required for conventional supervised fine-tuning [9, 10] . Gathering and labelling image segmentation data can be labour-intensive, requires expert approval, and is costly when acquiring data in the Polar regions due to their remote locality [11, 12] . Therefore, the Segment Anything Model (SAM) and the recently released Segment Anything Model 2 (SAM 2) from Meta FAIR provide researchers with the first segmentation foundation models [13, 14] . SAM and SAM 2 are pre-trained on large-scale image datasets with self-supervised learning but their applicability and performance on more specialist downstream applications for the segmentation of fractures on glaciers known as crevasses, is underexplored. This work explores the performance and generalisability of the SAM and SAM 2 foundation models when applied to centimetre resolution UAV images captured over a fast-flowing glacier in Svalbard using automatic and manual prompting without additional fine-tuning. The SAM and SAM 2 image segmentation architecture is included in Figure 1 .Melting and retreating glaciers worldwide have garnered significant interest as they directly contribute to rising sea levels [15] . Climate change and rising sea levels are causing concern around the world because of the disruption and devastating effects on coastal habitats [16] . In this context, understanding better the controls on glacier mass balance, which directly measures the amount of ice being lost, will help scientists analyse glacier dynamics in more detail. In the Arctic, which has been warming four times faster than anywhere else on the globe since 1979, Svalbard has lost ice cover at a rate of -10.5 Gt / yr in the early 21st century [17, 18] . According to [15] a total of 7563 ± 699 Gt of ice has been lost from ice sheets between 1992 and 2020.Crevasses on the surface of glaciers provide the line of weakness through which icebergs detach from tidewater glacier termini in a process known as iceberg calving [19, 20] . High rates of iceberg calving lead to glacier mass loss, hence it follows that increased crevassing may lead to further calving and hence mass loss. Furthermore, crevasses also play an essential role in regulating ice flow by acting as a conduit for surface meltwater to reach the bed and speed up ice flow [19] . However, manually mapping crevasses with Geographic Information System (GIS) software is laborious and time-consuming because of the the high level of skill required to accurately annotate crevasses. Therefore, automatically annotating crevasses using deep learning foundation models provides a solution to a labour-intensive process. However, like other remote sensing applications such as building or sea ice detection, crevasse segmentation usually involves complex scenes not found in everyday images, making it harder for algorithms to segment [21] .Using a foundation model to prompt an image and output a segmentation mask could help with starting the crevasse mapping process for a glacier or provide enough detail to give experts enough information to discover new findings. Currently, to date, a segmentation model that requires or does not require any prompting to segment crevasses to this standard is not available off the shelf. Therefore, the deep learning community and glaciologists in particular, could use the SAM or SAM 2 models built into an end-to-end user interface system to start the labelling process for segmentation to reduce labelling time and the amount of labour required.The contributions of this paper are summarised as follows. UAV images that are an optical image data modality have been compared in performance using the SAM and SAM 2 foundation models for image segmentation. The binary segmentation mask output from the SAM and SAM 2 models have been compared visually and with evaluation performance metrics for image segmentation applications. We also provide suggestions for future work that could improve the output segmentation results from the SAM 2 foundation model in the form of few-shot or weakly supervised learning.2 Related Work 
glaciology deep learning: 
 In recent years, automated algorithms have been explored because of how labour-intensive it is to manually map crevasses using GIS software [23] . The U-Net model was initially used to extract crevasses from the entire Antarctic Ice Shelf using Mean of Angles (MOA) images, demonstrating the use of deep learning for crevasse extraction [24, 25] . In [26] the U-Net model was enhanced by pre-processing the input Sentinel-1 Synthetic Aperture Radar (SAR) images from Antarctica with the Probabilistic-Patch Based filter to highlight crevasses in the SAR imagery. Further data augmentations were applied to increase the size of the dataset to train the U-Net model [26] . A Dice Score Coefficient (DSC) of 0.7602 was achieved with this method. However, the U-Net model failed to segment crevasses on the glacier where ridges or raised land passes underneath the ice, creating curvature on the surface. A further two models were proposed in [23] to distinguish between crevasses on Ice Shelves and Ice Sheets from Antarctica with the help of the proposed Parallel-Structure filter. Two types of crevasses had to be distinguished with each of the two models because of the angle of the radar signal often not capturing finer crevasses that are only a single pixel wide [23] . Therefore, thinner crevasses were eliminated to allow the overall algorithm to concentrate on larger crevasses that are often involved with glacier calving. The Re-sUNet model that adds residual connections to a standard U-Net architecture was proposed by [27] to further enhance the performance of crevasse identification where a DSC of 0.771 was achieved over 0.751 when using a standard U-Net architecture. In [20] MobileViT was used as a lightweight backbone feature extractor fine-tuned on SAR imagery from the West Antarctic Ice Sheet (WAIS). The backbone ViT was pre-trained on ImageNet-21K and fine-tuned on ImageNet-1K to increase the size of the pre-training datasets [28, 29] . Pre-trained backbone models trained on well-constructed datasets such as ImageNet as a feature extractor allow deep learning segmentation models to be adapted using less data than training the overall model from scratch. A DSC of 0.840 was achieved, improving the performance of Antarctic crevasse detection without any pre-processing steps for the input images. All of the above techniques have been applied to Antarctic crevasse segmentation rather than in the Arctic.The reasons for this are that crevasses in Antarctica are larger and appear on satellite imagery that is openly available for machine/deep learning research.As crevasses in the Arctic are smaller and harder to detect than in Antarctica, they are more challenging to segment for computerised algorithms.The SAM and SAM 2 foundation models released in 2023 and 2024 respectively by Meta FAIR are the first foundation models for image and video segmentation [13, 14] . The SAM models provided researchers with an open-source foundation model to use for segmentation applications with prompting. The main differences between the SAM models are that the SAM 2 model has smaller, more efficient ViT backbone models that are based on the Masked Auto Encoder Hiera (MAE-Hiera) models [30] , SAM 2 can be used on image and video data and the overall SAM 2 model has been trained on the SA-1B and SA-V datasets [13, 14] . In total, the SAM 2 model has been trained on one billion patches from eleven million images from the SA-1B dataset and 35.5 million video frames as image patches from the SA-V dataset. Overall, this is a total of 1.355 billion image patches. Therefore, the SAM 2 model is not only more lightweight in architecture but is the segmentation foundation model that has been trained on the largest quantity of image data available to date. However, as SAM models have recorded excellent performance metrics on general AI benchmark datasets that are in a similar domain to the training data of the SA-1B and SA-V datasets, it is challenging to adapt them to complex imagery such as for EO or medical applications. In [31, 32] SAM was used to segment glaciological features across the EO platforms from Sentinel  
uav data: 
 The spatial resolution of satellite imagery is much coarser compared to those acquired from UAVs, enabling mapping of finer scale structures on a glacier surface [33] . In this study, we used UAV imagery collected over the highly crevassed glacier terminus region in Svalbard [22] . The images were extracted with a variety of crevasse patterns to test the performance of the SAM models in a range of conditions. Structure from Motion (SfM) was used to construct an orthomosaic of the surveyed region at 10 cm pixel spacing from images captured on 8th August 2023 [34] . Figure 2 displays an example of the best UAV images segmented from the SAM and SAM 2 models that were captured from the Borebreen glacier in Svalbard, Norway.The UAV images were captured on 8th August 2023 because Svalbard in the Arctic is in the summer season. Capturing UAV image data in the summer months allows for maximum use of 24-hour daylight conditions in the Arctic as opposed to permanent darkness in the winter. Permanent darkness would prevent optical imagery from being captured because optical image sensors require ultraviolet (UV) rays from the sun to operate. UAV images are better in resolution than satellite imagery captured from space because UAVs fly 20 meters above the ground. Another limitation when using optical satellite imagery from space is that if there is any cloud cover or snow on the ground, it is captured with the image. Therefore, these limitations are removed by using UAV optical image data during summer because they fly below the cloud level, and there is less snow on the ground in the Arctic. The resolution at 10 cm per pixel in the UAV images provides a better resolution than 3 to 10 meters per pixel from satellite images captured from space. Therefore, the crevasses are larger in UAV imagery, making them less challenging to segment for computerised algorithms.The UAV images used for the experiments were cropped from the larger orthomosaic image in the GIS software package QGIS. The output image crops were converted from GeoTiff to RGB format and cropped to a resolution of 1024 x 1024 pixels each using the RasterIO package available to Python before modelling. An image resolution of 1024 x 1024 has been used to prevent images from being interpolated up to a larger resolution that is required to align with the input dimensions of the SAM model. Therefore, as much information in the image can be retained to improve the performance of the SAM model and allow a fairer evaluation against SAM 2. In total, ten images were converted to RGB format and cropped to allow them to be manually annotated for segmentation in the Computer Vision Annotation Tool (CVAT). Each of the ten annotated images was approved by an expert glaciologist to ensure proper evaluation of any modelling. 
mask generator experiments: 
 The SAM and SAM 2 model experiments have been carried out in the same way by using the automated mask generator and manual point prompt Python source code provided in the official GitHub repository for each model [35, 36] . Using the mask gen- erator to generate prompts on the UAV images automatically is a way to prompt the SAM and SAM 2 models without a human placing point or box prompts on the input image. Therefore, making it faster to apply prompts to each image tested during the evaluation of any modelling. The SAM and SAM 2 models are designed to segment objects in images [13, 14] . Therefore, as the ice on the glacier in the foreground is larger than the crevasses, the ice is prompted by the mask generator module. The mask generator from SAM and SAM 2 outputs a binary segmentation mask for each object in an image as an instance. Therefore, each segmentation mask for each instance must be combined into one overall binary segmentation mask in preparation for evaluation. The crevasses are located in the UAV imagery in the background, but the SAM and SAM 2 models are segmenting the glacier's ice in the foreground. Therefore, each segmentation mask output from the model must have the zero and one pixel values reversed to display the segmentation of the crevasses on the output mask. The converted segmentation masks output from the SAM and SAM 2 models were evaluated visually and with performance metrics for image segmentation. The DCE and Intersection over Union (IoU) metrics have been used because of the imbalance between the foreground and background pixel classes. There are more foreground pixels in the UAV imagery than in the background after the output segmentation masks have their zero and one values reversed.Metric SAM-B SAM-L SAM-H SAM2-T SAM2-S SAM2Further experiments were run with the SAM and SAM 2 models by applying a single background and foreground point prompt to the UAV images from Borebreen. The manual point prompt experiments were run using the single-mask and multi-mask settings on both models. The single-mask and multimask binary segmentation results were evaluated visually and with performance metrics for image segmentation. For the same pixel class imbalance reasons as the mask generator module experiments, the DCE and IoU performance metrics were used. However, instead of reversing the zero and one pixel values in each segmentation mask output, the background (red) and foreground (green) point prompts were reversed. A single background (red) point prompt was placed on the glacier's ice, and a single foreground (green) point prompt was placed inside a crevasse in the UAV images evaluated. Placing the point prompts in this order prompts the SAM and SAM 2 models to segment the crevasses on the glacier without needing to reverse the output segmentation mask's zero and one pixel class values after modelling. 
mask generator prompting: 
 After running tests with the SAM and SAM 2 mask generator modules, the automated prompting performance was evaluated on the ten UAV images. Ten test images have been used because of how labourintensive and time-consuming it is to annotate images and gain expert approval. In the time frame for writing the paper, all the labelled images for semantic segmentation have been used in the experiments. However, ten 1024 x 1024 resolution UAV images are the same as having 160, 256 x 256 images which is a common resolution used for deep learning computer vision experiments. The performance evaluation results are included in Table 1 , which show that the SAM 2 Hiera-L model outperforms the other SAM models with average DSC and IoU scores of 0.43 and 0.28, respectively. However, when inspecting the segmentation masks from each model, it was noticed that both models detected false positives around the areas of the segmented crevasses. The walls inside some of the crevasses were found to have pixel values similar to those of the glacier's surface, which the SAM models are mistaken for crevasses. The SAM models were also noted to detect false negatives because the dark pixel colours inside the crevasses are the same colour as the sediment that grows on the glacier surface outside the crevasses. The SAM 2 Hiera-L model outperformed the SAM ViT-H model because the visual results display that they align with the SAM 2 Hiera-L model's average DSC and IoU performance metrics in Table 1 The average DSC and IoU scores were analysed with a line plot that includes a dust line to display the average DSC and IoU values and the overall average DSC and IoU values for the complete range between models in Figure 3 . The line plots were used to visually display the DSC and IoU model performance metrics against one another and their data distributions for the ten UAV images in the test dataset. The dust line plots show that the SAM 2 Hiera-L performs the best and has a data distribution over a smaller range than the other SAM models except for image 6. Image 6 is an outlier because it is very challenging for the SAM models to segment crevasses because of how small the crevasses are in the image compared to the other nine images evaluated. Therefore, all of the performance evaluation results and plots indicate that the SAM 2 Hiera-L model performs the best because it has been trained on the largest pre-training dataset and has the largest model architecture of the SAM 2 models.After evaluating the performance of the automated prompting experiments using the SAM and SAM 2 mask generator modules, manual point prompt experiments were evaluated. The evaluation test results from the single-mask and the best performing multi-mask output for each SAM and SAM 2 model have been presented in Tables 2 and 3 respectively. Table 2 shows that the SAM ViT-H model outperforms the other SAM and SAM 2 models when applying point prompts in the single-mask mode with average DCE and IoU values of 0.15 and 0.08. However, after evaluating the best overall performance from the three segmentation masks output from the multi-mask mode for the SAM and SAM 2 models in Table 3 , it was found that the SAM 2 Hiera-L and SAM ViT-H performs almost on par with the SAM ViT-L model. The SAM ViT-L performed the best during multi-mask mode point prompt experiments with DCE and IoU values of 0.21 and 0.12. The dust line plots display the reduced DSC and IoU results over a larger data distribution range for the single and multi-mask experiments with the SAM and SAM 2 models over the mask generator results. The dust line plots are included in Figures A. 21 and A.22 in the appendix. The SAM models generate confidence scores through a combination of mask prediction, refinement using attention mechanisms and the conversion of logits to probabilities using the sigmoid activation function. Therefore, the confidence scores generated in the multi-mask setting of the SAM models help guide the model to a better output perdition. A better output prediction is made because three masks are generated and scored to reflect the model's certainty about each mask generated instead of one in the single-mask mode. Using the multi-mask mode of the SAM models helps limit the adverse effects on the model from ambiguity, complex shapes, overlapping objects, and noisy data. Overall, the manual point prompt experiments with both single-mask and multi-mask modes set for the SAM and SAM 2 models did not outperform the segmentation results of the SAM 2 Hiera-L model using the mask generator module's automated prompting. 
conclusion & future work: 
 All of the available SAM and SAM 2 models have been evaluated on ten UAV images from the Borebreen glacier. The SAM 2 Hiera-L model was found to perform better during the evaluation of this work for automated prompting experiments than the other SAM and SAM 2 models. During the automated prompting experiments, the SAM 2 Hiera-L model outperformed all SAM and SAM 2 models run with single and multi-mask point prompt tests. This displays that foundation models that are the size of the SAM 2 Hiera-L architecture and trained on the same quantity of image data has the potential to be used on challenging out-of-domain images for EO segmentation applications. However, as the DSC and IoU performance metrics are lower than required for glaciology experts to analyse the model outputs, further fine-tuning techniques are required to be implemented to increase model performance. Finetuning is required because of how far out of the domain the UAV images are from the images in the SA-1B and SA-V training datasets for the SAM and SAM 2 foundation models. In future work fewshot and weakly supervised learning are techniques that can be used for finetuning to overcome data shortages and labelling challenges. Depth estimation, image classification and object detection for crevasses could also be included in future works.  
 
 Appendix:  


Decision: Overall, the authors have successfully improved the paper by incorporating the reviewers' feedback, enhancing its clarity for the reader. Based on the reviewers' comments on the authors' rebuttal and my own assessment of the paper, I recommend accepting this paper as a poster presentation.

Reviews: review: title: title: A well-written manusciprt
paper_summary: paper_summary: This paper studies drone images of crevasses in Arctic, which is an important but less explored domain. Authors apply SAM and SAM models to segment crevasses and qualitively evaluated the resutls, which suggests that further fine-tuning is still essential for using machine learning models on the crevasses segmentation task.
main_review: main_review: My concerns on experiment settings are resolved, and the size of the dataset is not a main issue due to the difficulty of collecting these dsata. The quantitative results is also very inclusive.
However, the utilization of SAM models is insufficient and page limit is not a relevant reason. I encourage authors to explore and report more details on the results when adjusting the models, including image sizes, image conditions (such as the density of crevasses shown in the image), number of prompt points, format of prompts (such as words or bounding boxes), internal parameters of the models (such as thresholds).
strength_weakness: strength_weakness: - It seems like the full power of SAM models are not fully elicited.
  - According to section 4, authors use SAM models to conduct inference in two situations: 1. grid points prompt 2. two single points prompt of background and foreground.
  - According to Figure 2, grid points for the images may result in point being located in both the crevasse region and non-crevasse region, which will make the segmentation results worse.
  - Considering the image resolution and the distribution of crevasses in the images, it woule be neccessary to adjust the sparsity of the grid points prompts.

- For figure 2, descriptions are missing why the resolution 1024x1024 is chosen. The ground truth of crevasse has quite complex shapes in the current resolution. Considering SA-1B and SA-V datasets are mostly about common objects, decress the dron image resolution to have simpler shape crevasses may help.

- (Please forgive me on lacking relevent knowledge.) Five images seem like too few for a valid evalution. Difficulites on collecting data (like cost and time span) is not described in the manuscript.

- Further discussion on using SAM models to relieve the burden of annotation can be appended.
title: title: Application of Segment Anything (foundation) models for detecting crevasses in UAV imagery
paper_summary: paper_summary: The paper aims to quantify the generalization ability of the foundation models known as SAM (Segment anything Model) and SAM 2 on UAV imagery of arctic glaciers for the purpose of detecting crevasses. 
Without any baselines, the paper compares several variants of SAM and SAM2 and employs the Dice score (DSC) and intersection-over-union (IOU) as metrics for performance. The paper concludes that the best-performing model is not good enough for downstream applications and fine-tuning is required.
main_review: main_review: Thank you for the rebuttal. However, I am maintaining my score because of the weak experimental evaluation as my concern about basics, such as proper metrics, relevant baselines etc. must be included to help put the results in perspective and therefore baselines cannot be left for future work.
strength_weakness: strength_weakness: The only novelty of the paper is the application of SAM and SAM2 to UAV data, but the paper provides very little details about the applied methodology. Despite the use of off-the-shelf models, it seems like it would be very hard to reproduce the results. There are no details about how each of the models, prompts, etc. are configured. 

Moreover, the evaluation of the model is also quite weak. First of all, there are no baseline models included as a reference. If the authors had provided the performance metrics for simple baselines as predicting the majority class or random guessing or a stronger baseline such as a trained U-NET (even for the small dataset they have) would help put the reported performance metrics into perspective. The simple baselines are really relevant since several of the models seem to perform very poorly (i.e. IoC as low as 0.03).

The choice of metrics also seems a bit odd. The paper uses IoU and DSC, which are essentially quantifying the same thing (you can write one as a function of the other). The similarity of the metrics is also very evident from Figure 3. It would have been much more informative to include other metrics or use the space for elaborating on the methodology. Finally, it is not clear to me what the purpose of the analysis of Figure 3. 

Minor details: The abbreviation 'DSC' does not seem to be defined, but I assume it is the Dice score.