Bringing SAM to new heights: leveraging elevation data for tree crown segmentation from drone imagery
Authors: Mélisande Teng
Arthur Ouaknine
Etienne Laliberté
Yoshua Bengio
David Rolnick
Hugo Larochelle
Venue: NeurIPS 2025 poster ()
Keywords: Forest monitoring
Remote sensing
Instance segmentation
Segment Anything Model
Drone imagery
Climate change
Earth observation
Primary area: applications

Abstract: Information on trees at the individual level is crucial for monitoring forest ecosystems and planning forest management.  Current monitoring methods involve ground measurements, requiring extensive cost, time and labour. Advances in drone remote sensing and computer vision offer great potential for mapping individual trees from aerial imagery at broad-scale. Large pre-trained vision models, such as the Segment Anything Model (SAM), represent a particularly compelling choice given limited labeled data. In this work, we compare methods leveraging SAM for the task of automatic tree crown instance segmentation in high resolution drone imagery in three use cases: 1) boreal plantations, 2) temperate forests, and 3) tropical forests. We also look into integrating elevation data into models, in the form of Digital Surface Model (DSM) information, which can readily be obtained at no additional cost from RGB drone imagery.  We present BalSAM, a model leveraging SAM and DSM information, which shows potential over other methods, particularly in the context of plantations. We find that methods using SAM out-of-the-box do not outperform a custom Mask R-CNN, even with well-designed prompts. However, efficiently tuning SAM further and integrating DSM information are both promising avenues for tree crown instance segmentation models.

Bringing SAM to new heights: Leveraging elevation data for tree crown segmentation from drone imagery
Mélisande Teng 1.2
Arthur Ouaknine 1,3,4
Etienne Laliberté²
Yoshua Bengio
David Rolnick 1,3
Hugo Larochelle
Mila Québec Al Institute
2 Université de Montréal
McGill University
"Rubisco AI
tengmeli @mi la quebec
Abstract
Information trees at the individual level is crucial for monitoring forest ecosys- tems and planning forest management. Current monitoring methods involve ground measurements, requiring extensive cost, time and labor. Advances in drone remote sensing and computer vision offer great potential for mapping individual trees from aerial imagery at broad-scale. Large pre-trained vision models, such the Segment Anything Model (SAM), represent a particularly compelling choice given limited labeled data. In this work, we compare methods leveraging SAM for the task of automatic tree crown instance segmentation in high resolution drone imagery in three cases: 1) boreal plantations, 2) temperate forests and 3) tropical forests, We also study the integration of clevation data into models, in the form of Digital Surface Model (DSM) information, which can readily be obtained at no additional cost from RGB drone imagery. We present BalSAM, model leveraging SAM and DSM information, which shows potential over other methods, particularly in the context of plantations. We find that methods using SAM out-of-the-bor do not outperform custom Mask R- R-CNN, even with well-designed prompts. However, efficiently tuning SAM end-to-end and integrating DSM information are both promising avenues for tree crown instance segmentation models.
Introduction
Data on individual trees are important for understanding forest ecosystems and supporting sustainable forest management. Such data are essential, for example, to answer questions about forest compo- sition, tree growth, and tree health and mortality. They are also particularly relevant in the context of biodiversity assessments natural climate solutions, in measuring the carbon stored in forests and evaluating the success of afforestation, reforestation and revegetation policies [1, 2]. Specifically, access species identity and individual tree crown delineation data is crucial, as different tree species have different allometries [3, 4, 5]. Indeed, carbon stored in a tree can be recovered with allometries using information about the crown surface area, the species and the height of the tree [6].
Individual trees are still largely monitored by conducting ground surveys [7], requiring extensive cost, time and labour. However, recent advances in deep learning, alongside the decreasing cost of drones with tree crown delineation. Popular deep learning methods, such Mask R-CNN [8] and Retina [9], have been extensively used in the context of vegetation monitoring using remote sensing data, but they most often do not focus identifying tree species [10, 11]. Despite the success of deep learning methods for tree mapping at scale using remote sensing imagery [12. 13], instance segmentation of tree crowns remains understudied, large part because of the lack of annotated data at the individual tree level.
cameras, open up possibilities for automatically performing individual
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In contexts where task-specific data are not abundant, practitioners often resort to pre- pre-trained models from large datasets. The Segment Anything Model (SAM) [14], for example, designed to segment any object in an image either in zero-shot setting or when given prompts in the form of points, boxes, masks or text, SAM has been used out-of-the-box for a wide variety of applications, such as medical imaging [15] and river water segmentation from remote sensing imagery [16]. However, despite its zero-shot capabilities, SAM has been found to perform poorly in certain segmentation tasks when used directly in its automatic mode [17] and, consequently, number of methods have been developed to adapt SAM to specific tasks without requiring that be fine-tuned fully [18, 19]. In particular, RSPrompter [20] proposed to learn how to generate appropriate prompts for SAM in order to segment objects of interest in remote sensing imagery. Keeping the image encoder and mask decoder frozen, a learnable prompter taking input the image embeddings from the image encoder is trained to produce task-relevant prompts for the mask decoder.
The integration of task-specific information from the Digital Surface Model (DSM) into tree crown instance segmentation models has also been underexplored. The DSM provides a surface elevation map including above-ground objects, and is a product that is always readily available at additional cost from the drone RGB imagery acquisition. Indeed, Structure- Structure-from-Motioe from-Motion (SIM) photogrammetry, which is used to create RGB orthomosaics from high-resolution drone imagery, generates 3D photogrammetry dense point clouds, from which the DSM is derived. Thus, the DSM provides complementary 3D structural information without additional data collection overhead.
In this work, we assess the potential of SAM and the value of auxiliary DSM data for the problem of tree crown instance segmentation from high-resolution drone imagery, through three realistic use cases: boreal plantations, temperate forests and tropical forests. We introduce BaISAM, model building on RSPrompter that allows SAM to incorporate DSM information through parameter-efficient prompt learning. We evaluate the effectiveness of BaISAM, as compared to SAM's automatic mode and RSPrompter. Our study highlights the limitations of SAM in its intended use as an out-of-the-bon and user-friendly tool. However, we find that methods that learn task-specific prompts in a module integrated to SAM outperform custom-traines CNN models. We also find that integrating DSM representations within SAM or CNN-based approaches generally improves model performances for tree crown instance segmentation, with the benefits being dependent the structural complexity of the canopy.
In summary, our contributions are: 1) assessing SAM's capacities for tree crown instance segmentation from high-resolution drone imagery, 2) introducing new methods leveraging the DSM within both SAM-based and convolutional architectures, and 3) analyzing the performances of these methods across three different forest types. This work proposes the first benchmark of instance segmentation methods on the Quebec Plantations [21], Quebec Trees [22] and BCI [23) datasets. We release project code at https:/ gi thub. cwm/ http://eelsanddeeeee saadetenggBalll...
2 Related work
Tree segmentation Recent advances in remote sensing and machine learning have enabled the mapping of trees at scale, including both detection and semantic segmentation tasks [11, 24, 10]. However, many ecological use cases (e.g. monitoring phenology, biomass, and species distributions) require fine-grained information on tree species and crown size, calling for instance segmentation of tree crowns by species. This task has remained understudied due to the limited availability of labelled high-resolution datasets. Brandt et al. [25] and Tucker et al. [13] successfully mapped individual trees from satellite imagery, but insufficient resolution hindered classification of tree species. In works considering tree segmentation with higher resolution data [26, 27, 28], the majority either do not classify trees or consider only a limited set of classes. Such works [29, 30, 31, 32, 33] typically rely on popular architectures such as Mask R-CNN [8] and U-Net [34], though several studies propose modified versions of Mask R-CNN to segment and classify individual tree crowns [35, 36, 37] and Firoze et al. [32] explore advanced transformer-basee architectures. Classical computer vision [38, 39, 40] and machine learning [41, 42] approaches have also been explored.
Algorithms incorporating tree height data Canopy height maps (CHM) derived from airborne or drone LiDAR laser returns provide complementary structural information to 2D RGB imagery and have previously been estimated [43, 44, 45, 46, 47, 48] or integrated [49] in methods developed for satellite and drone [36, 30, 50] remote sensing data. Pixel-based approaches from classical computer

---

vision such as watershed segmentation, region-growing and edge detection [51, 52, 53] have been used on CHM data [54] for individual tree crown delineation. However, these methods often rely on rules and careful parameter tuning, making it challenging to use them in multi-species contexts. CHM information has also been explored for individual tree crown segmentation including relying on a custom Mask R-CNN architecture [36, 55], using the CHM as an additional input channel to a Mask R-CNN [56] or directly using raw LiDAR with point cloud-based approaches [30]. While CHMs derived from LiDAR offer high structural resolution, the digital surface model (DSM) produced by photogrammetry avoids the cost of specialized sensors and more readily aligned with drone imagery. In addition, the DSM from photogrammetry gives equally accurate 3D surface representation of forest canopies as LiDAR [57]. Schiefer et al. [58] found using DSM information alongside RGB drone imagery to be a promising avenue for the task of semantic segmentation of trees in drone imagery, but this work did not tackle the task of instance segmentation.
SAM in Earth observation Foundation models for computer vision offer promising avenues for Earth observation tasks. In particular, the Segment Anything Model (SAM) [14] has achieved effective visual segmentation in images across range of use cases. Several methods for adapting SAM to Earth observation have been proposed, including delineating crop field boundaries [59], classifying land cover [60] and identifying urban villages [61]. Khazaie and Wang [19] proposed toolkit to adapt SAM to custom datasets and applied it for semantic segmentation of trees in satellite imagery. Osco et al. [18] proposed a method based on SAM, using text prompts segment instances of a given class. However, the method requires iterative updates which would be computation and time intensive when many instances are present an image. Further, the pre-trained text prompt encoder could be limited in its ability to capture fine- tne-grained classes, such as different tree species. Grondin et al. [62] trained detector to better prompt SAM to segment tree trunks from ground level imagery. but did not consider classification of species. Chen et al. [20] proposed RSPrompter, method that learns how to generate appropriate prompts to SAM, to segment objects of interest in remote sensing imagery (see Section 4.
In this work, we aim use the DSM to improve RGB-based tree crown instance segmentation, as well as enabling SAM to leverage the DSM by building upon insights from RSPrompter [20]. To our knowledge, SAM has neither been used to segment and classify individual tree crowns, nor been leveraged with height information.
3 Datasets
We compare methods on three datasets repre- senting different realistic application contexts: boreal plantations, temperate forests and tropical forests. As we discuss further in Section 5, each case presents different data characteristics. Plan- tations (created for timber production carbon sequestration) typically consist of trees planted in orderly rows around the same time, while forests do not, as shown in Figure 1. In this sec- tion, we present each dataset and detail the data pre-processing. Further details are presented in Appendix A.
Image
Annotations
Normalized
alantation
Quebec Plantations dataset We use RGB orthomosaics. photogrammetry digital surface models (DSMs), tree crown delincation and species labels in plantation sites from the UAV Canadian (Quebec) Plantations dataset [21]. The imagery has resolution of 5 mm/pixel. We exclude the Serpentinl and Serpentin2 sites from our study because they contain respectively only 25 and 39 annotated trees and keep 15 sites of interest. We consider tree species that have more than 20 trees across all sites, and group the remaining species into an "Other" category, resulting in total of 9 classes. The annotations corre-
Figure 1: Examples of the raw image, annotations and DSM (normalized for the purpose of visualiza- tion) on each of the datasets under consideration.
3

---

spond to the plantations' trees, but other trees may be visible in the imagery e.g. , trees outside plantation's 's area on the border of the orthomosaic. We manually delineated areas of interest (AOIs) in QGIS to exclude trees that do not have corresponding annotation in the imagery. We split the data spatially into training, validation and test sets, defining polygonal regions corresponding to geographical blocks to avoid spatial autocorrelation, and we ensure that each class is represented in all sets. Orthomosaics are either assigned entirely to a split or assigned to different splits by manually delineating areas in QGIS. We detail further the splitting strategy in Appendix A.I.
SBL dataset We consider the Quebec Trees dataset [22] which covers temperate forest site and use the RGB imagery and corresponding DSM from date 2021-09-02, for which 22, 933 tree crowns were manually labelled. In this paper, we refer it as SBL dataset, for Station de Biologie de Laurentides, the site where the imagery was collected, to avoid confusion with the Quebec Plantations dataset. The resolution of the imagery 1.9 cm/pixel. We use the AOIs defined in Ramesh et [63] for training, validation and testing. Since annotations are not always available at the species level, we consider 18 classes of interest 11 tree species, 4 genera, families, a class corresponding to dead trees and an "Other" class.
BCI dataset We use the 2022 imagery of the Barro Colorado Island crown maps dataset [23], covering a 50-ha rectangular plot of tropical forest at a resolution of 4 cm/pixel with corresponding "improved version" of the crown map data. This version contains 1 112 species with 2, 280 tree crown delineations that were obtained by manually delineating tree crowns and further refining them with SAM with human supervision. The corresponding DSM is provided imagery in 8-bit encoding. therefore at m-height resolution. As noted by Vasquez et al. [23], there missing annotations from undetected tree crowns. We manually correct for missing annotations by masking out parts of the imagery that contain unannotated trees. Given the large number of species, the long-tailed distribution and challenging nature of fine- fine-grained classification of trees in this context, we group the trees by taxonomic family. Due to the low number of instances in certain classes and the spatial split avoid geospatial auto-correlation, we further group certain families into an "Other" class so that all families are represented in the training and test sets, leaving 31 classes of interest.
fourth channel to the
Pre-processing We use the geodataset vO.2.2 Python package divide the orthomosaics into 1024 x 1024 tiles with 50% overlap. We exclude tiles without labels and tiles with more than 80% black pixels at the border of the AOIs. We also exclude annotations where less than 20% of the tree appears in the tile. We detail class codes, corresponding scientific names and the number of trees per class for the different datasets in Appendix A, as well as details on the composition of the train, validation and test splits.
4 Methods
We extensively study the performance of SAM and the informativeness of the DSM for tree crown instance segmentation. We compare different methods, including models with the DSM used as input along with the RGB imagery and present several ablations and variations of our main methods. We detail choices of backbones and hyperparameters in Section 4.3 and Appendix B.6.
4.1 Methods description
SAM out-of-the-box We first assess what extent SAM can segment tree crowns in our dataset without additional training or tuning. We benchmark SAM in the automatic mask generation mode (denoted SAM). Following classical approaches such as watershed segmentation, we also test the use of local maxima of the DSM, potentially corresponding to treetops in the RGB image, to prompt SAM (denoted SAM+DDM prompts). Further details on this method in Appendix B.2. An overview of SAM+DSM prompts is shown in Figure 6 along with sample images and prompts from each dataset. For both models we apply Non-Maximum Suppression (NMS) on the segmented instances. We also considered using the DSM image as a dense prompt, but obtained very poor segmentation masks, as dense prompts are intended to be binary masks (see Appendix B.3).
lntrps:/hhugobaudccon. htw.://wwccbcoogt.tt..oooooaaaad wtw.i//ggooaasaeeeeoo html

---

image embedding
RSPrompter
prompt encoder
Image
SAM image encoder
SAM
mask decoder
DSM
Predictions
DSM encoder
frozen
truinable
Figure 2: Overview of our BaISAM method.
Mask R-CNN and variations We consider Mask R-CNN comparison, since this architecture has previously been successfully used for tree crown instance segmentation on aerial imagery [35, 36, 11, 37]. We compare Mask R-CNN trained from scratch and initialized with weights from model pre-trained on ImageNet. We also consider an additional variant that stacks the DSM with the RGB input fourth channel (Mask R- R-CNN+DSM
Faster/Mask R-CNN+SAN and variations Motivated by SAM's high quality segmentation when given human input prompts, we consider training models to predict boxes and masks in an attempt to better prompt SAM and refine the predictions. We train Faster R-CNN for tree crown detection on each dataset to provide box prompts SAM (Faster CNN++AMM This corresponds to the SAM-det method presented in Chen et al. [20]. We also train Mask R-CNN on each dataset to prompt SAM with predicted boxes and/or masks (Mask SAMM Similarly to the Mask R-CNN baseline, we also consider stacking the DSM modality to its corresponding RGB image as fourth channel for both Faster R-CNN+ + SAM and Mask S-C.
Mask2Former We include comparisons with the transformer- architecture Mask2Former [64], which was developed for universal image segmentation tasks. We fine-tune Mask2Former models that pre-trained on the COCO instance segmentation dataset [65], using the Swin-base Swin-L backbone, and using RGB or RGB+DSM as input, stacking the DSM to its corresponding RGB image fourth channel.
RSPrompter The size of SAM makes it challenging to fully fine-tune it on small datasets. SAM's image encoder has 632M parameters and fine-tuning SAM would require considerable compute re- sources, rendering its training process inaccessible to most forest-monitringg practitioners. Therefore, we consider lightweight methods leveraging components of SAM without requiring full fine- -uninng We leverage RSPrompter [20] in our study, as this method was originally developed specifically for instance segmentation tasks in remote sensing imagery. We choose the RSPrompter-anchh version, as the architecture of the prompter is built on Faster R-CNN, and other methods in this benchmark are R- CNN-baddd. We train it following Chen et al. [20].
BaISAM We propose method leveraging RSPrompter by integrating DSM embeddings to enhance SAM image representations. Our method, named BaISAM (in reference to the tree species balsam fir) aims at learning better prompt SAM thanks height information and canopy structures captured by the DSM modality. Since SAM was designed to use dense prompts with binary segmentation masks on top of point prompts, we investigate whether integrating the DSM RSPrompter a similar way helps guiding the segmentation and the classification. We introduce a trainable DSM encoder module fuse DSM and image embeddings with an element-wise sum. This global embedding is then fed as input to the SAM decoder, similarly to dense prompts. An overview of BalSAM is provided in Fig. 2.
4.2 Evaluation
We evaluate our instance segmentation models with mean Average Precision (mAP). Given the class imbalance in our dataset, we also consider a weighted mAP (wmAP), where the weights are defined
5

---

Single-class
Multi-class
Model
DSM
Pre-trained
mAP
mloU
MAP
wmAP
SAM (100 pps)
K
8.05
35.06
SAM (10 pps) SAM
X
10.11
34.01
V (promptss)
11.17
50.91
X
x
59.36 #0.12
79.63 =0.31
42.69
41.63
55.75 +0.87
Mask R-CNN
X
63.65 10.25
81.82 10.21
46,51
10.65
58.30
10.71
V
/
64.64 10. 40
81.89
10.35
48.96
10.61
60.32
10.42
X
x
53.56 1.0.12
76.22
10.12
33,52
10.25
45.79
10.39
Faster R-CNN+SAM
X
57.85 20.38
78.00
=0.32
39,79
10.68
50.30
+0.87
/
/
58.00 1:0.14
78.27
4:044
40.14 10.81
52.08
1:1000
Mask R-CNN+ SAM
X
57.60 10.11
78.18
10.18
39,76 10.69
50,46 10.30
/
57.83 +0.06
77.65
=:029
41.13 +0.65
51.33 +0.49
Mask2Former (Swin-base)
X
54.01 0.70
69.80
0.85
33,900 0.39
42.24 0,58
/
7
58.56 =:0.5
72.95
=:021
37.36 4: 1,16
47.15
1:583
Mask2Former (Swin-L)
X
61.77.10.3 0.39
73.41
0.38
44,331 0.38
52.92
10049
5
61.43.100 0.4
73.38
0.17
41.17
1 1.045
51.72. 1.05
RSPrompter BaISAM
X
66.37 10.53
82.58
10.94
52,77
10.59
62.37
11.41
65.03 =1.01
83.24
=0.24
54,40
#2.31
64.84 +0.86
Table Results on the Quebec Plantations test dataset, averaged over seeds, All metrics are multiplied by 102
and reported with standard errors. The column Pre-trained refers to ImageNet pre-training for the backbones of the Mask R-CNN, Faster R-CNN and Fask2Former models (SAM is always pre-trained); "_" denotes not applicable. We bold and underline the best and second best scores.
by the proportion of examples of each class in the test Since SAM out-of-the-box provides segmentation masks each instance but no associated class label, we also evaluate the models with mAP considering the single class "trees". Finally, (mloU) with the single class "trees", by matching each ground truth instance to the predicted instance with the highest associated loU. We then average IoU scores that mIoU does not reflect false positive instances, as it only compares each ground truth instance with single predicted instance namely, the best matching one in terms of IoU. This metric reflects only the quality of the segmentation if the object has been correctly detected in a setting where we only consider single class for all trees.
consider the mean Intersection over Union
all instances in the dataset. Note
4.3 Implementation details
In all experiments, we use the ViT-Huge version of SAM. For the Faster R-CNN+DSM and Mask R-CNN+DSM methods, we initialize the ResNet-50 backbone of Faster
R- CNN+DSM with ImageNet weights. To allow for stacking the DSM to the image input, we randomly initialize the first layer allow for 4 input channels. Then, copy back the ImageNet pre-trained backbone's weights of the first layer onto the RGB channels. For all trained models, we apply RandomFlip augmentations during training and normalize the DSM by its maximum value per sample. We select the best model based on the validation segmentation mAP value (over all classes). For BalSAM, the DSM encoder follows the architecture of the dense prompt encoder in SAM and is a 3-layer CNN with layer normalization and GeLU activation. We provide further details on training hyperparameters and model architectures App. B.6. Our methods are all trained on a single GPU with 24GB CPU memory and 48GB GPU memory.
5 Results
Tables 1, 2 and summarize the model performances in terms of single-class "tree" metrics and aggregated mAP metrics over the classes for each dataset. We report per class mAP performance Appendix C. The BCI dataset the most challenging setting as it consists of a large number of classes with high visual similarity. Therefore, for this dataset, we only compared the methods that were most competitive on the Quebec Plantations and SBI. datasets. We also show examples of predictions from different models in Figure 3.

---

Single-class
Multi-class
Model
DSM
Pre-trained
mAP
mloU
MAP
WMAP
SAM (100 Pps)
6.56
35.70
SAM (10 pps)
X
5.63
21.19
SAM
V (prompts)
8.24
41.90
X
X
26.16
+0.33
60.07
#0.80
19.10
20.23
22.45
=0.22
Mask R-CNN
X
/
32.44 10.12
65.08
21.38
10.17
27.27
=0.18
/
32.37 10.18
64.08
20.17
20.87
40.13
26.82
=0.15
Faster R S-CNN+SAM
/
27.38 1:0.13
61.40
20.11
19.72
40.10
23.23
=0.06
/
28.00
+:009
.94
20.20
20.52
40.10
23.89
=0.08
Mask R- R-CNN+SAM
X
26,21
#0.17
61.67
#0.36
18.23
+0.17
21.83
=0.19
25.94
#0.12
61.19
#0.17
17.73
+0.14
21.36
=0.30
RSPrompter BalSAM
X
33.59
+1.02
64.25
24.94
10.52
29.44
=0.83
33.55
+0.93
66.02
11.49
24.88
+0.63
29.12 =0.81
Table 2: Results on the SBL test dataset, averaged over 3 seeds. All metrics are multiplied by 102 and reported with standard emrors The column Pre-trained refers to ImageNet pre-training for the backbones of the Mask R-CNN and Faster R-CNN models (SAM always pre-trained); "_" denotes not applicable. We bold and underline the best and second best scores.
Single-class
Multi-class
Model
DSM
Pre-trained
mAP
mlnU
MAP
wmAP
SAM (100 pps) SAM (10 pps)
8.19
43.13
X
7.01
28.51
SAM
(promptts)
11.86
59.76
Mask R-CNN
X
30.39
#0.82
61.74
#0.16
5.52 +0.01
10.33 =0.27
/
31.93 10.41
10.79 63.38
6,34 10.02
10.50
Mask R-CNN + DSM encoder
/
32.62
10.69
63,20
10.08
8,30
10.29
11.86
20.27
RSPrompter BalSAM
35,55
70.76
60.72
+0.85
8.44 40.13
11.53
#0.34
34.66
10.30
61.60
12.32
8.48 10.29
10.42 10.27
Table 3: Results the BCT test dataset, averaged over 3 seeds. All metrics multiplied by 102 and reported with standard errors. The column Pre-trained refers to ImageNet pre-training for backhones the Mask R-CNN models (SAM is always pre-trained); "_" denotes not applicable. We bold and underline the hest and second best scores.
5.1 Discussion
Overall, we find that RSPrompter and BalSAM perform better than Mask R-CNN methods and that including the DSM as additional input information improves predictions. In the following, we prioritize wmAP to assess the performance of the fodels-for those that can be evaluated with class-wise mAp-as
P-as our datasets have significantly unbalanced classes.
Using SAM out-of-the-box is suboptimal, even with carefully designed prompts. Qualitatively, we observe that in many cases, SAM automatic fails to separate overlapping crowns into separate masks and confidently segments the background tiny plants, leading many false positives. It also misses trees in areas where tall herbaceous vegetation occurs. We show qualitative results in Fig. and Fig. 5 (App. B. We find that SAM+DSM. in which SAM is prompted with local maxima in the DSM, is only somewhat more performant. When a prompt corresponding to an overall treetop is given, SAM is generally able to correctly segment the tree crown, explaining the modest boost in mloU compared to SAM automatic. However, local maxima corresponding to small plants or different parts of a single tree crown can be given as prompts the mask decoder shown in Fig. (App. B), often leading to false positives.
Interestingly, prompting SAM with boxes masks output by trained Mask R-CNN degrades performance compared to the predictions that same trained Mask R-CNN. We observe that SAM sometimes focuses on very small details and artifacts in the imagery, degrading the quality of the original segmentation. Qualitative results are shown in Figure (Appendix B.4). Similarly, we find that Faster R-CNN+SAM models perform significantly worse than Mask R-CNN.
Initializing R-CNN backbones with pre-trained ImageNet weights helps. Mask R-CNN competitive on all datasets, and initializing the ResNet-50 backbone with ImageNet weights of Mask R-CNN improves performance, compared to training from scratch. We make the same observation with the Faster R-CNN backbone of the Faster SAM method.
7

---

State-of-the-art Mask2Former does not outperform Mask R-CNN While we find that using RGB+DSM improves performance compared to using RGB alone as input, Mask2Former baselines (using Swin-base or Swin-I. b do not outperform Mask R-CNN on the task of tree crown instance segmentation on the Quebec Plantations dataset (Table 1), despite their higher capacity. This is in line with prior works that have highlighted limitations of Mask2Former in the context of forest monitoring from remote sensing imagery [66, 63].
Methods learning to prompt SAM end-to-end outperform the other methods. RSPrompter and BalSAM models outperform Mask R- R-CNN-basse models (integrating or not the DSM) in terms of multi-class mAP and wmAP on all three datasets. We show qualitative results of our models' predictions on the Quebec Plantations and BCI datasets in Figure 3. Looking at class-wise metries, we also find that RSPrompter and BalSAM generally perform significantly better than Mask R-CNN- based methods on less prevalent classes on the Quebec Plantations and SBI. datasets (Table 11 in Appendix C.1 and Tables 12 and 13 in Appendix C.2).
Integrating the DSM can improve predictions, but challenges remain for classification in dense forests with many species. Importantly, we observe that the benefit of using the DSM is highly dependent on the structure of forested area. Intuitively, the DSM is relevant for two main reasons: (1) it captures the vertical structure of individual trees which can improve classification, (2) it represents the spatial structure of trees relative to one another, which can improve segmentation. The Quebec Plantations dataset, where the DSM impact is the greatest and most consistent across methods, is composed of well-separated young trees with visible ground. The SBL and BCI datasets are more challenging, both in terms of classification and segmentation, given the larger number of classes, overlapping tree crowns and noisy annotations. In the dense, closed canopies of the SBL. dataset, individual trees hardly stand out in the DSM, as can be seen in Figure 1. The DSM is thus less informative, and models integrating the DSM perform comparably to their counterpart without DSM. We demonstrate this numerically by training Mask R-CNN with only the DSM as input (no RGB imagery) on the Plantations and SBL datasets. We see a much larger drop in mloU on the SBL dataset when comparing to Mask R-CNN models using image inputs, while on the Plantations dataset, mloU remains high (see Table 14 Appendix D.1).
In the tropical forest of the BCI dataset, there are large differences between tree heights and structures, even with dense and closed canopies. Adding the DSM information improves predictions of Mask -CNN-based models on the BCI dataset, even though it is only available a coarse m-vertical resolution. We additionally report the performance of a model encoding the DSM with a CNN module before stacking it the RGB image and passing it as input a Mask R-CNN (Mask R-CNN +DMM encoder in Table 3), and find that adding capacity to process the DSM information can improve further on Mask R-CNN-DSM showing great potential for future work. We provide implementation details in Appendix D.3.
Class wise analysis on the Quebec Plantations dataset reveals patterns aligned with challenges known to ecologists. Looking more closely into the per-class performance for instance segmentation models (i.e. excluding SAM out-of-the-Bee thox based methods) the Quebec Plantations dataset, observe performance generally increases with the number of examples for a given class, as shown in Fig. 11 (App. C.1). However, all methods perform relatively well on Acer saccharum (acsa), despite there being few examples of this class, which can be attributed to this class having very different visual features than the rest of the species. The performance of the different models differs most on the Picea mariana (pima) class. In fact, it is a very similar species to the most common class in our dataset, Picea glauca (pigl). In ground field surveys, these two species are most easily distinguished by looking at the shapes of the cones rather than characteristics visible in drone imagery. In our models, incorrect classifications of Picea mariana tend to be for Picea glauca (Fig. 12 in App. C.I).
5.2
Ablation studies
We test several ablations and variations of our main methods using the SBL and BCI datasets.
Mask R-CNN prompts to SAM SAM can be prompted with both dense prompts in the form of binary masks and point prompts in the form of bounding boxes, points or text. We compare using Mask R-CNN output segmentation masks, detection boxes, or both as prompts to SAM. We find that feeding masks only yields poorer results. Additionally, computation of the mAP metric requires scores which usually correspond to detection scores for predicted boxes. We compare using boxes
8

---

Grand truth
SAM
SAM:DSM prompts
Mask R-CNN
Mask R-CNN-DOO
RSPrompter
BaISA
Quiherr Plantations
BCI
Figure 3: Qualitative results comparing methods presented Sec. on the Quebec Plantations and BCI test sets. Samples were chosen at random in the test set. For the SAM and SAM+DSM prompts columns, colours do not correspond to particular classes since SAM does not classify instances. Colours in other columns correspond classes and consistent across columns. BaISAM is able to produce higher quality segmentations following more closely the shape of tree crowns, methods integrating the DSM produce fewer misclassifications.
scores from Mask R-CNN, masks scores from SAM or the average of both for the computation of the mAP. While we do not observe significant impact on performance for different choices on the Quebec Plantations dataset, the best performance achieved for box prompts only with toxes--masks scores on the SBL dataset (see Table 15 in Appendix D.2).
Incorporating DSM information in Mask R-CNN Observing that Mask R R-CNN+DSM does not perform significantly better than Mask R- R-CNN on the SBL dataset, we explore other ways of including the DSM. We use the DSM vertical and horizontal gradient maps two additional channels stacked to the image input. We also consider adding capacity in the Faster R-CNN module of Mask R-CNN by adding an extra fully connected layer to the bounding box predictor and the classification head. We do not observe significance improvement in the performance as reported Table (App. D.3). Finally, we test the effect of encoding the DSM before combining it with the image first processing DSM through CNN and stacking the DSM embedding to the image as fourth channel before passing to Mask R- R-CNN. We provide more details about these models in App. D.3.
Losses The SBL dataset classes are highly imbalanced and we compare three losses with the standard cross-entropy used in our experiments: 1) a weighted cross-ent cross-entropy loss using the inverse frequency of class occurrences in the training set as weights and 2) hierarchical loss based on the trees taxonomy, which is weighted sum of loss at the species, genus and family level. We define this loss in Appendix D,4.1, 3) focal loss [9]. The gamma parameter was set to 2 for the focal loss. We find that the weighted cross entropy yields poor performance, due the high-class imbalance, and that the model trained with focal loss does not perform as well as the cross-entropy loss. We also find that the hicrarchical loss does not significantly improve performance compared to the regular cross-entropy setup (see Table 17 in Appendix D.5).
Additional post- proc-sron The default NMS in Mask R-CNN is not class agnostic, as it removes overlapping predictions only if they have the same class. Unlike in autonomous driving datasets, on which Mask R-CNN is often used and pre- pre-trained, we do not encounter occlusions in our dataset and we expect only one object to be visible at given location. Therefore, we consider a class-agnostic NMS, but do not observe significant improvements on the SBL or BCI datasets. This is likely because the chosen metrics favour having multiple candidate predictions for an instance - including the correct. label, even if it does not have the highest score over missing the correct class entirely.

---

Variations on BalSAM We consider variations on how the DSM information is integrated into BaISAM. We first consider version in which the prompt encoder receives the encoded DSM added to the image embedding as input, instead of the image embedding alone. Second, we consider a modified setup in which the mask decoder receives DSM information only through the prompt encoder. We evaluate these methods on the BCI dataset, but do not observe significant improvements from the original BaISAM model. We detail these variations in Appendix B.5.
5.3 Recommendations
Our study shows that using the DSM along with the RGB imagery consistently improves segmentation and classification results for the plantation use case. Therefore, we recommend that practitioners looking to quantify the carbon stored in boreal plantations include the DSM information in their models. We leave it to practitioners to decide, based on their application and available data, which models to use. For example, if only bounding box annotations are available, we showed that Faster R-CNN+SAM is a reasonable bascline, and that including the DSM helped. We also report the inference speed and the number of trainable parameters of different models in Table 10. Tropical forests remain a difficult case, with known challenges related to obtaining ground truth species labels, and to the structural and spectral similarity of forests of different taxonomic composition [67]. This poses the broader question of framing a task that meets user needs and is feasible in tropical forests, where individual tree carbon mapping might not be possible. For example, as the largest trees store the vast majority of forest carbon [68]. a first step for carbon estimation in tropical forests could be to focus on large trees only, which might reduce the complexity in the number of species.
6 Conclusion
In this work, we investigate the potential of SAM for tree crown instance segmentation from high- resolution drone imagery, considering the settings of tree plantations, boreal forests and tropical forests. We show that methods using SAM out-of-the-box, even with well designed prompts, are suboptimal compared to the widely used architecture Mask R-CNN. However, we find that methods that learn to prompt SAM through further tuning are promising for this task. Finally, we also demonstrate that using DSM information can improve predictions. With the growing number of available drone imagery datasets for forest monitoring, the release of DSM data alongside orthomosaics may be low-hanging fruit, as such data can be obtained directly from RGB imagery.
We highlight several limitations of the present work. On the methodological side, we find that while RSPrompter and AM demonstrate superior performance to other methods, they also show higher variance. Our work does not fully address the classification challenges associated with long-tailed training data (beyond experiments with hierarchical, weighted and focal losses); further exploration through e.g. class rebalancing could improve performance. On the application level, we note that users building on this work should demonstrate care in regard to potential dual uses, such as risks associated with the release of models trained to identify species commonly targeted in illegal logging.
We hope our work will help advance the impactful use of machine learning in biodiversity protection and nature-based climate solutions, via improved tools for forest monitoring. Promising future directions include exploring different architectures for the DSM encoder of BalSAM, improving the methods' robustness (e.g. stabilising the training process with regularization and augmentations), and evaluating the effectiveness of different methods in a low-data regime or few-shot setting. Indeed, in practice, experts might be able to provide few manual labels of species of interest. This work opens the door to other ways of using height information, for instance, predicting DSM as an auxiliary task rather than using the DSM as an input, or using the 3D point clouds obtained from SIM directly. Using a depth or canopy height map model, could also be explored. Another potential direction is adding contextual metadata into our models in the form of spatial or spectral priors, or the type of forest, to improve the classification performance. Indeed, location metadata could prove helpful, as shown in species distribution modelling contexts [69]. Spectral information, available through e.g. satellite open-source programs, could also be added as additional input signal into the models. Additionally, while we have so far trained models separately on each dataset, as more high-resolution drone imagery data becomes available, learning representations on combined datasets at different resolutions could provide a foundation for models that can generalize to local contexts and trees species. Developing casily adaptable methods to different forest ecosystems has considerable potential for impact.
10

---

References
[1] Josep G Canadell and Michael R Raupach. Managing forests for climate change mitigation. 2008.
science, (456588))1146-111
[2] Alice Di Sacco, Kate A Hardwick, David Blakesley, Pedro HS Brancalion, Elinor Breman, Loic Cecilio Rebola, Susan Chomba, Kingsley Dixon, Stephen Elliott, Godfrey Ruyonga, et al. Ten golden rules for reforestation to optimize carbon sequestration, biodiversity recovery and livelihood benefits. Global Change Biology, 27(7):1328 10---145, 1348, 2021.
[3] Vishal Singh, Ashish Tewari, Satya PS Kushwaha, and Vinay Dadhwal. Formulating allometric equations for estimating biomass and carbon stock in small diameter trees. Forest Ecology and Management, (61(11):1945-111 19449999999,,, 2011.
[4] Damena Edae Daba and Teshome Soromessa. The accuracy of species-specific allometric equations for estimating aboveground biomass in tropical moist montane forests: case study of Albizia grandibracteata and Trichilia dregeana. Carbon bulance and management, 14:1-13,
2019.
[5] Abu Mulatu, Mesele Negash, and Zerihun Asrat. Species-specifie allometric models for reducing uncertainty estimating above ground biomass Moist Evergreen Afromontane forest of Ethiopia. Scientific Reports, 14(1):114) 2024.
[6] Tommaso Jucker, Fabian Jörg Fischer, Jérôme Chave, David A Coomes, John Caspersen, Arshad Ali, Grace Jopaul Loubota Panzou, Ted R Feldpausch, Daniel Falster, Vladimir A Usoltsev, et al. Tallo: A global tree allometry and crown architecture database. Global change biology, 28(17):525--5268, 2022.
[7] Verra. v1.0, 2023.
Reforestation, and Revegetation URL //verr/ hwtp//errr.rr.meeeeeeeeeeeee www.methodologoeeooooooooooooooo vm0047- wtw.://reeeetttttoeeeeeetttaaaaa and awt.aaaao.....a--eeeeeeeoooooooo 2025-02-08.
Afforestation,
and-
Accessed:
[8] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R- R-CNN. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017. 1,2
[9] Lin. Focal loss for dense object detection. arXiv preprint urXiiv:7086 02002, 2017. 1,9
[10] Ben G Weinstein, Sergio Marconi, Stephanie Bohlman, Alina Zare, and Ethan White. Individual tree-crown detection in RGB imagery using semi-supervised deep learning neural networks. Remote Sensing, 11(11):130% 2019. 1,2
[11] James GC Ball, Sebastian HM Hickman, Tobias D Jackson, Xian Jing Koay, James Hirst, William Jay, Matthew Archer, Mélaine Subry-KKietz,, Grégoire Vincent, and David A Coomes. Accurate delineation of individual tree crowns in tropical forests from aerial RGB imagery using Mask R-CNN. Remote Sensing in Ecology and Conservation, 95):641 2023. 1,2,
[12] Gyri Reiersen, David Dao. Bjöm Lütjens, Konstantin Klemmer, Kenza Amara, Attila Steinegger, Ce Zhang, and Xiaoxiang Zhu. ReforesTree: A dataset for estimating tropical forest carbon stock with deep learning and aerial imagery. In Proceedings of the AAAI Conference on Artificial 2022.
Intelligence, volume 36, pages
[13] Compton Tucker, Martin Brandt, Pierre Hiernaux, Ankit Kariryaa, Kjeld Rasmussen, Jennifer Small, Christian Igel, Florian Reiner, Katherine Melocik, Jesse Meyer, et al. Sub-continental- scale carbon stocks of individual trees in African drylands. Nature, (15(7950):80 86, 2023. 1,
2
[14] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment Anything, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026,
2023. 2.3
[15] Dongjie Cheng, Ziyuan Qin, Zekun Jiang, Shaoting Zhang, Qicheng Lao, and Kang Li. SAM on medical images: A comprehensive study on three prompt modes. arXiv preprint arXiv:2305.00035, 2023.
[16] Armin Moghimi, Mario Welzel, Turgay Celik, and Torsten Schlurmann. A comparative performance analysis of popular deep learning models and Segment Anything Model (SAM) for river water segmentation in close-range remote sensing imagery. IEEE Access, 2024. 2
11

---

[17] Tianrun Chen, Lanyun Zhu, Chaotao Ding, Runlong Cao, Yan Wang, Zejian Li, Lingyun Sun, Papa Mao, and Ying Zang. SAM fails to segment mnything?-SAAM LAM-adapters Adapting SAM in underperformed scenes: Camouflage, shadow, medical image segmentation, and more. arXiv preprint arXiv:2304.001 091444 2023. 2
[18] Lucas Prado Osco, Qiusheng Wu, Eduardo Lopes de Lemos, Wesley Nunes Gonçalves, Ana Paula Marques Ramos, Jonathan Li, and José Marcato Junior. The Segment Anything Model (SAM) for remote sensing applications: From zero to one shot. International Journal of Applied Earth Observation and Geoinformationn 124: 2023. 2,
[19]
Vahid Reza Khazaie and Marshall Wang- Segmate Python segmentation toolkit, 2023. URL https:// thub. Coom/Vectorio wwwoorIInttttttttssssaaaa
Python package. 2,
[20] Keyan Chen, Chenyang Liu, Hao Chen, Haotian Zhang, Wenyuan Li, Zhengxia Zou, and Zhenwei Shi. RSPrompter: Learning to prompt for remote sensing instance segmentation based on visual foundation model. IEEE Transactions Geoscience and Remote Sensing, 2024. 2, 3,
2,, 22, 23,31 31
[21] I. Lefebvre and E. Laliberté. UAV LiDAR, UAV Imagery, tree segmentations and ground measurements for estimating tree biomass in Canadian (Quebec) plantations, 2024. URL
https://doi.org: 10. 20383/ 103..0030000 0979. 2, 3
[22] Myriam Cloutier, Mickaël Germain, and Etienne Laliberté. Influence of temperate forest autumn leaf phenology on segmentation of tree species from UAV imagery using deep learning, Remote Sensing of Enviromment, 311. 311:114283, 2024. 2, 4, 17
[23] Vincente Vasquez, Katherine Cushman, Pablo Ramos. Cecilia Williamson, Paulino Villareal, Luisa Fernanda Gomez Correa, and Helen Muller-L Barro Colorado Island 50-ha plot crown maps: manually segmented and instance segmented. (version 2). smithsonian tropical research institute., 2023. 2.
[24] Irem Ulku, Erdem Akagündüz, and Pedram Ghamisi. Deep semantic segmentation of trees using multispectral images. IEEE Journal of Selected Topics Applied Earth Observations and Remote Sensing, 0675889-6044, 2022. 2
[25] Martin Brandt, Compton J Tucker, Ankit Kariryaa, Kjeld Rasmussen, Christin Abel, Jennifer Small, Jerome Chave, Laura Vang Rasmussen, Pierre Hiernaux, Abdoul Aziz Diouf, et al. An unexpectedly large count of trees in the West African Sahara and Sahel. Nature, 587(7832):
78-82, 2020. 2
[26] Weijia Li, Haohuan Fu, Le Yu, and Arthur Cracknell. Deep Learning Based Oil Palm Tree Detection and Counting for Righ--eesolution Remote Sensing Images. Remote Sensing, 91):22, December 2016. ISSN 2072-4292. doi: 10.3390/r99010022. URL. httpt://ww/ww mdpi. com/ 0072-4292/9/1222
[27] Teja Kattenborn, Jens Leitloff, Felix Schiefer, and Stefan Hinz. Review on Convolutional Neural Networks (CNN) in vegetation remote sensing. ISPRS Journal of Photogrammetry and Remote Sensing, 173:24-49 March 2021. ISSN 09242716. doi: 10. isopsioss.ssrrsppsss... 2020. 12.010. URL
https://linkingee
elsevier cww www.://rerrriieeaa////////222210
2
[28] Masanori Onishi and Takeshi Ise. Explainable identification and mapping of trees using UAV RGB image and deep learning. Scientific Reports, 11(())903, January 2021. ISSN 2045- 2322. doi:
URL https: / nature. com/ aarricestcce
841598- 020-9633----- 9. 2
[29] Kunyong Yu, Zhenbang Hao, Christopher J Post, Elena A Mikhailova, Lili Lin, Gejin Zhao, Shangfeng Tian, and Jian Liu. Comparison of classical methods and Mask R-CNN for automatic tree detection and mapping using uav imagery. Remote Sensing, 14(2):295, 2022. 2
[30] Sebastian Dersch, Alfred Schoett, Peter Krzystek, and Marco Heurich. Towards complete tree crown delineation by instance segmentation with Mask R-CNN and DETR using UbVebased multispectral imagery and lidar data. ISPRS Open Journal of Photogrammetry and Remote Sensing, 8:100037, 2023. 2,
[31] Andrew J Chadwick, Nicholas C Coops, Christopher W Bater, Lee A Martens, and Barry White. Transferability of a Mask R-CNN model for the delineation and classification of two species of regenerating tree crowns to untrained sites. Science of Remote Sensing, 9: 1000109, 2024. 2
12

---

[32] Adnan Firoze, Cameron Wingren, Raymond A Yeh, Bedrich Benes, and Daniel Aliaga. Tree in- stance seginnntttio with temporal contour graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2193-2202, 2023. 2
[33] Maximilian Freudenberg, Paul Magdon, and Nils Nölke. Individual tree crown delineation in high-resolution remote sensing images based on U-Net. Neural Computing and Applications, 34124))22117-22227,, 2022. 2
[34] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCA/ 2015: 18th international conference, Munich, Germany, October 5-9,
2015, proceedings, part III 18, pages 234-241. Springer, 2015. 2
[35] Chong Zhang, Jiawei Zhou, Huiwen Wang, Tianyi Tan, Mengchen Cui, Zilu Huang, Pei Wang. and Li Zhang. Multi-species individual tree segmentation and identification based on improved Mask R-CNN and UAV imagery in mixed forests. Remote Sensing, 14(4):874, 2022. 2,
[36] Yingbo Li, Guoqi Chai, Yueting Wang, Lingting Lei, and Xiaoli Zhang. ACE R-CNN: An Attention Complementary and Edge Pitection-Basee Instance Segmentation Algorithm for Individual Tree Species Identification Using UAV RGB Images and LiDAR Data. Remote Sensing, 14(13)33035, June 2022. ISSN 2072-4292. doi: 10.33900/s144133000 URL https: www..ddp.... 2ttp//272-42200 21////4133 1111411/30000 13/3035. 3,5
[37] Maojia Gong. Weili Kou, Ning Lu, Yue Chen, Yongke Sun, Hongyan Lai, Banggian Chen, Juan Wang. and Chao Li. Individual Tree AGB Estimation of Malania oleifera Based on UAV-RGB Imagery and Mask R-CNN. Forests, 14(7):1493, July 2023. ISSN 1999-4907. doi:
00739///14071999
URI. https::///// :/tlpps://// //wp:w http:://wwwwwwp.....p..tpp ctw.dpp..omme.m (01999999999944111
2,5
[38] François A. Gougeon. A Crown-Following Approach to the Automatic Delincation of Individual Tree Crowns in High Spatial Resolution Aerial Images. Canadian Journal of Remote Sensing, 2143):774-84,, August 1995. ISSN 0703-8992, 1712-7971. doi: 10. 111110000738999221999999 199.9998462999008876229099 1080/07038992 1995 10874622.
URL http::/wwwt tandf ine. com/doi/ ht////ddiaabb/// ab.
[39] Tomas Brandtberg and Fredrik Walter. Automated delineation of individual tree crowns in high spatial resolution aerial images by multiple-scale analysis. Machine Vision and Applications, 11 (2):64-73, October 1998. ISSN 0932-8092, 1432-1769, doi: 00.1007/00013880500001 URL
http://link. http:///innk.springer com 10. 0007/8001300050911 2
[40] Darius Culvenor. TIDA: an algorithm for the delineation of tree crowns in high spatial resolu- tion remotely sensed imagery. Computers & Geosciences, 28(1):33-44, February 2002. ISSN 00983004, doi: 10.
10-2.
e elsevier.
[41] Mats Erikson. Species classification of individually segmented tree crowns in high- high-resolution aerial images using radiometric and morphologic image measures. Remote Sensing of Environ- ment, 91(3-4):469 477, June 2004. ISSN 00344257. doi: 10.10166).rse.2004.444,
URL
https://linking:: wwwpeeiinnuuu.l.eeeeeeeeeeeeeeee elsevier...
2
[42] Yinghai Ke and Lindi Quackenbush. A review of methods for automatic individual tree- crown detection and delincation from passive remote sensing. International Journal of Remote Sensing, 32(17):4725-4777 September 2011. ISSN 0143-1161, 1366-5901. doi: URL https: //t///www// /www. tandf onl htwp///eeeoeeeeeeeeeeeeeee.eeeee htt///oi/fffff /do/fflfff/fooooooo 1080/01431161 2010. 09110..4948.... 2
[43] Jamie Tolan, Hung-I Yang, Benjamin Nosarzewski, Guillaume Couairon, Huy V. Vo, John Brandt, Justine Spore, Sayantan Majumdar, Daniel Haziza, Janaki Vamaraju, Theo Moutakanni, Piotr Bojanowski, Tracy Johns, Brian White, Tobias Tiecke, and Camille Couprie. Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar. Remote Sensing of Environment, 300: 10000:13888 38888, January 2024. ISSN 00344257. doi: 10.16 13888. URL https://1inkinghh 2
elsevier . com/ wwt./ooo//ttieeeee http:/ettreeeeeeeeee wwweeeeeeeiiieee /ttp:///ttteeee/////////////0330
[44] Jan Pauls, Max Zimmer, Una M. Kelly, Martin Schwartz, Sassan Saatchi, Philippe Ciais, Sebas- tian Pokutta, Martin Brandt, and Fabian Gieseke. Estimating Canopy Height at Scale. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine
13

---

Learning, volume 235 of Proceedings of Machine Learning Research, pages 39972-39988. PMLR, July 2024. URL https://proceedding mlr wwtpa/eeesee///paleeeeeee./
html.
[45] Fabien H. Wagner, Sophia Roberts, Alison L.. Ritz, Griffin Carter, Ricardo Dalagnol, Samuel Favrichon, Mayumi C.M. Hirye, Martin Brandt, Philippe Ciais, and Sassan Saatchi. Sub- meter tree height mapping of California using acrial images and LiDAR-informa U-Net model. Remote Sensing of Environment, 305:114099, May 2024. ISSN 00344257. doi: elsevier con/retrieve/pii/
(0.1016/f.rse.2024,, 14099. 14099, 500344257240011000 2
URL
[46] Fabien H. Wagner, Ricardo Dalagnol, Griffin Carter, Mayumi CM Hirye, Shivraj Gill, Le Bi- enfaiteur Sagang Takougoum, Samuel Favrichon, Michael Keller, Jean PHB Ometto, Lorena Alves, Cynthia Creze, Stephanie P. George-Chacon, Shuang Li, Zhihua Liu, Adugna Mullissa, Yan Yang. Erone G. Santos, Sarah R. Worden, Martin Brandt, Philippe Ciais, Stephen C. Hagen, and Sassan Saatchi. High Resolution Tree Height Mapping of the Amazon For- est using Planet NICFI Images and http: / www.i..og/abb.....o.ooo
U-Net Model, January 2025. URL 10600. arXiv:2501.1060 [cs]. 2
[47] Manuel Weber, Carly Beneke, and Clyde Wheeler. Unified Deep Learning Model for Global Prediction of Aboveground Biomass, Canopy Height, and Cover from High-Resolution, Multi- Sensor Satellite Imagery. Remote Sensing, 17(9): April 2025. ISSN 2072-4292. doi: (.3390/ss17099550 URL https:
htttp::/wwww... /tww:/wwwww...o mdpi cow www/2072--oe 2111//4492/////9/
[48] Tony Chang, Kiarie Ndegwa, Andreas Gros, Vincent A. Landau, Luke J. Zachmann, Bogdan State, Mitchell A. Gritts, Colton W. Miller, Nathan F. Rutenbeck, Scott Conway, and Guy Bayes. VibrantVS: A High-Resole Righ--eesolution Vision Transformer for Forest Canopy Height Estimation. Remote Sensing, 1716) March 2025. ISSN 2072-4292. doi: 10.3390/rs170611 7061017. URL https://wwv. /ttp::/wwwwmmpp....m wttpcoommococcccc http:/227222222/200 /1//////1////// //107. 2
[49] Martijn Vermeer, Jacob Alexander Hay, David Völgyes, Zsófia Koma, Johannes Breidenbach, and Daniele Stefano Maria Fantin. LiDAR-based Norwegian tree species detection using deep learning, November 2023. URL http: http:///axvvv.oree wwweiivorrg//ba wttgg/as/////.o..e
arXiv:2311.06066
[cs]. 2
[50] Binbin Xiang, Maciej Wielgosz, Theodora Kontogianni, Torben Peters, Stefano Puliti, Rasmus Astrup, and Konrad Schindler. Automated forest inventory: Analysis of high-density airborne LiDAR point clouds with 3D deep learning. Remote Sensing of Environment, 305: 14078, URL attts://linkknge:
May 2024. ISSN 00344257. doi: 10.10166j.se..2244.111. 2
elsevier
[51] Le Wang, Peng Gong, and Gregory S Biging. Individual tree-crown delineation and treetop detection in rigsosio aerial imagery. Photogrummetric Engineering & Remote Sensing, 70(3):351-333 -577, 2004. 3
[52] Hongyu Huang, Xu Li. and Chongeheng Chen. Individual tree crown detection and delincation from very-high-resolutinn UAV images based on bias field and segmentation algorithms. IEEE Journal of selected topics in applied earth observations and remote sensing, 11(7):2253-201 2262, 2018.
watershed
[53] Zhenyu Ma, Yong Pang, Di Wang, Xiaojun Liang, Bowei Chen, Hao Lu, Holger Weinacker, and Barbara Koch. Individual tree crown segmentation of larch plantation using airborne laser scanning data based on region growing and canopy morphology features. Remote Sensing, 12 (7):1078, 2020. 3
[54] Mojdeh Miraki, Hormoz Sohrabi, Parviz Fatehi, and Mathias Kneubuehler. Individual tree crown delineation from high-resolution UAV images in broadleaf forest. Ecological Informatics, 61:101207, 2021. 3
[55] Zhenbang Hao, Lili Lin, Christopher J Post, Elena A Mikhailova, Minghui Li, Yan Chen, Kunyong Yu, and Jian Liu. Automated tree-crown and height detection a young forest plantation using mask region-based convolutional neural network (Mask R- R-CNN). ISPRS Journal of Photogrammetry and Remote Sensing, 178: 172:112, 2021. 3
[56] Sizhuo Li, Martin Brandt, Rasmus Fensholt, Ankit Kariryaa, Christian Igel, Fabian Gieseke, Thomas Nord-Larsen, Stefan Oehmcke, Ask Holm Carlsen, Samuli Junttila, et al. Deep learning enables image-based tree counting, crown segmentation, and height prediction at national scale. PNAS nexus, pgadd6,,, 2023.
14

---

[57] Giulio B. Santoro, Paulo G. Molin, José M. S. M. Viveiros, Giovanna de Andrade Ferreira. Vinicius M. Costa, Leo E. Haneda, Melodie K. S. D. Sinegalia, Laury Cullen Jr. Pedro H. S. Brancalion, Carlos A. Silva, and Danilo R. A. de Almeida. Monitoring the structure of restored forests and assessing aboveground carbon density through canopy metrics derived from digital aerial photogrammetry and LiDAR. Restoration Ecology, n/a(t/a))e70077. ISSN 1526-1000 doi: 10.111 3
[58] Felix Schiefer, Teja Kattenborn, Annett Frick, Julian Frey, Peter Schall, Barbara Koch, and Sebastian Schmidtlein. Mapping forest tree species in high resolution UAV-based RGB-imagery by means of convolutional neural networks. ISPRS Journal of Photogrammetrr and Remote Sensing. 170:205 215, December 2020. ISSN 09242716. doi: 10.1016/j 20.00.00........................ elsevier
URL https: /
3
[59] A SAM-based method for large-scale crop field boundary delineation, 2023. IEEE. 3
[60] Bowei Xue, Han Cheng, Qingqing Yang, Yi Wang, and Xiaoning He. Adapting Segment Anything Model to aerial land cover classification with low-rank adaptation. IEEE Geoscience and Remote Sensing Letters, 21:1-5, 2024.
[61] Xin Zhang, Yu Liu, Yuming Lin, Qingmin Liao, and Yong Li. UV-SAM: Adapting Segment Anything Model for urban village identification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22520-22528 2024. 3
[62] Vincent Grondin, Philippe Massicotte, Mohamed Gaha, François Pomerleau, and Philippe Giguère. Leveraging Prompt-Bases Basee Segmentation Models and Large Dataset to Improve Detection of Trees. Proceedings of the Conference Robots and Vision, may 28 2024. http://cc..puppub.og/uubbtttt..t
3
[63] Venkatesh Ramesh, Arthur Ouaknine, and David Rolnick. Tree semantic segmentation from aerial image time series. arXiv preprint #rXiv:2407.13102 2024. 4, 8, 17, 26
[64] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290-1299, 2022.
[65] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014. 5
[66] Joshua Veitch- Ceitch-Michaelis, Andrew Cottam, Daniella Schweizer, Eben Broadbent, David Dao, Ce Zhang, Angelica Almeyda Zambrano, and Simeon Max. OAM-TCD: A glob- ally diverse dataset of high-resolution tree cover maps. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL httpps//oppnrrevis..u wtt.:etteeooommmiitet23300000000
8
[67] Qi Yang, Maaike Y Bader, Guang Feng, Jialing Li. Dexu Zhang, and Wenxing Long. Mapping species assemblages of tropical forests at different hicrarchical levels based on multivariate regression trees. Forest Ecosystems, 10: 100120, 2023. 10
[68] James A Lutz, Tucker J Furniss, Daniel J Johnson, Stuart J Davies, David Allen, Alfonso Alonso, Kristina Muderson-Teixeire Ana Andrade, Jennifer Baltzer, Kendall ML Becker, et al. Global importance of large-diamd diame-ei trees. Global Ecology and Biogeography, 27(7):849 864, 2018. 10
[69] Johannes Dollinger, Philipp Brun, Vivien Sainte Fare Garnot, and Jan Dirk Wegner. Sat-sinr: High-resolution species distribution models through satellite imagery. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 10:41 2024. 10
Acknowledgments
We thank Isabelle Lefebvre, Guillaume Tougas, Anne-Marie Cousineau, Chloé Fiset and members of the LEFO lab for insightful discussions and sharing field work knowledge. We thank Hugo Baudchon for iterations on the geodataset package and helpful discussions. We also thank Francis Pelletier and the Mila IDT team for their support with the Mila compute infrastructure.
15

---

Funding statement This project was undertaken partially thanks to funding from IVADO, the Canada First Research Excellence Fund, and the Canada CIFAR AI Chairs program. This research was enabled in part by compute resources, software and technical help provided by Mila (mila.quebec).
A Dataset
In this section, we provide more details about the composition of the datasets and the splits that we used in this study.
A.1 Quebec Plantations dataset
We summarize the classes considered in our study in Table 4, break down the composition of each
site in the dataset in Table and present the distribution of species per split in Table 6.
piba pima
Pinus banksiana
Picea mariana
pist
Pinus strobus
pigl
Picea glauca
thoc
Thuya occidentalis
ulam
Ulmus americana
beal
Betula allegnaniensis
acsa
Acer Saccharum
other
Other, Larix laricina, Pinus resinosa, Populus tremu-
loides, Betula papyrifera,
Quercus rubra
Table 4: Species codes for considered classes and corresponding scientific names in the Quebec Plantations dataset.
piba
pima
pist
pigl
thoc
ulam
beal
acsa
other
total
cbpapinas
136
121
1437
182
142
5
32
2076
cbblackburnl
1440
215
102
100
0
0
1865
cbblackburn2
573
18
50
993
0
0
67
1702
cbblackburnå
0
0
140
3
0
O
2
145
cbblackburn4
278
0
355
125
0
775
bbblackburn5
86
0
0
514
0
0
0
3
603
cbblackburn6
3002
273
122
1746
216
149
0
5513
cbbernard]
0
O
221
0
0
0
14
235
cbbernard2
14
0
0
cbbernard3
283
377
531
7
2
8
73
19
1300
cbbernard4
0
206
1193
0
00000
0
0
1402
afcamoisan
0
0
0
628
0
0
0
2
630
afcahoule
0
1004
0
0
1005
afcagauthmelpin
0
0
0
0
0
1674
1674
afcagauthier
0
500
0
0
500
Total
5379
925
992
9079
763
419
23
78
1842
19500
Table 5: Number or trees species per site of the Quebec Plantations dataset.
The training, validation and test sets were defined such that all classes were represented in the training and train and test set. There are neither image pixels, nor annotations that belong to two different splits. While some sites are represented in both the testing and training sets in the Quebec Plantations dataset, we tried as much as possible assign sites fully to splits and limit spatial autocorrelation The only reason why some orthomosaics were divided into both the training and test sets was ensuring that species of interest were both in the train and test sets. This is the only time when we manually drew AOIs. However, much possible, the train/val/test regions were kept spatially separate
16

---

piba
pima
pist
pigl
thoc
ulam
acsa
beal
other
total
train
19869
1377
2224
32496
1079
709
179
51
3343
61327
val
6978
2046
2447
6710
573
245
116
40
3713
22868
test
1471
1056
544
6519
1946
1050
56
19
1601
14262
Table 6: Tree species annotations distribution in the different Quebec Plantations splits. Note the values for each set and species are higher than the number trees because tiles have 50% overlap.
Site
Train
Val
Test
cbpapinas
cbblackburnl
/
cbblackburn2
cbblackburn3
ebblackburns
ebblackburn5
cbblackburn6
cbbernard
cbbernard2
cbbernard3
1
ebbernard4
afcamoisan
afcahoule
afcagauthmelpin
afcagauthier
Table 7: Assignment of sites of the Quehec Plantations dataset to splits.
"blocks" ensuring no overlap between the splits. Table 7 shows site assignment to splits. Only the sites afccagauthmelpin and afcagauthier were split into the train and test sets.
A.2 SBL dataset
While Ramesh et al. [63] and Cloutier et al. [22] conducted previous studies on the SBL dataset in the context of semantic segmentation, some classes in the annotations were ignored. As much as possible, we group those classes into ones already considered in the study. Some annotations are only provided the genus family level. For example, classes of interest include Acer saccharum, Acer rubrum and Acer pensylvanicum. but some instances only have the label "Acer". We choose keep genus level classes as separate classes instead of grouping them all into an "Other" category as it would end being composed of many different species. Certain species only have very few instances and we group them into two supercategories "Pinopsida" and "Magnoliopsida" for conifers and non-conifers, which are also the level which some annotations are provided.
modify some of the classes used these studies, noting that
We summarize the classes we consider for the task on the SBL dataset, with the corresponding names in the original annotations in Table 8. We also show the number of instances per class per split in Table 9.
A.3
BCI dataset
The BCI dataset contains annotations for 2280 tree crowns covering 112 species. Given the long- tailed distribution of tree species and the need to split the orthomosaic spatially to avoid spatial nuto-correlation, we decide consider classes at the family level. We group further group some families into the "Other" class, such that all families are present in the train and test sets. We show distribution of trees from the BCI dataset family classes considered in our study in The "Other" class contains the following families: Clusiaceae, Polygonaceae, Malpighiaceae, Myr- taceae, Erythropalaceae, Vochysiaceae, Frythroxylaceae, Sapindaceae, Staphyleaceae, I Lythraceae, Elacocarpaccae, Rhizophoraceac, Monimiaccae, Violaccae, Solanaceae and Other.
17

---

Class
Corresponding annotation codes
Dead
Dead
Pinopsida
Conifere
Magnoliopsida
Feuillus, QURU (Quercus rubra L.), OSVI (Ostrya virginiana
(Mill.) K.Koch), PRPE (Prumus pensylvanica Lfil), FRNI (Frax-
tnus nigra Marshall)
Thuja occidentalis I..
THOC (Thuja occidentalis)
Abies balsamea (I..) Mill.
ABBA (Abies balsamea)
Larix laricina (Du Roi) K.Koch
LALA (Larix laricina)
Tsuga canadensis (L.)
TSCA (Tsuga canadensis)
Betula L.
Betula, BEPO (Betula populifolia Marshall)
Fagus grandifolia Ehrh. Populus L.
FAGR (Fagus grandifolia)
Populus, POBA (Populus balsamifera L.), POGR (Populus gran-
didentata Michx), POTR (Populus tremuloides Michx.)
Acer L.
Acer
Acer pensylvanicum L. Acer saccharum Marshall Acer rubrum L.
ACPE (Acer pensylvanicum)
ACSA (Acer saccharum)
ACRU (Acer rubrum)
Pinus strobus L.
PIST (Pinus strobus)
Betula alleghaniensis Britton
BEAL (Betula alleghaniensis)
Betula papyrifera Marshall
BEPA (Betula papyrifera)
Picea A.Dietr.
Picea, PIGL (Picea glauca (Moench) Voss), PIMA (Picea mari-
ana (Mill. Britton et al.), PIRU (Picea rubens Sarg.)
Table 8: Classes considered in the SBL dataset, as well as corresponding codes and scientific names in the original annotations. In orange classes at the family level, and in teal are classes the genus level.
Pinepiila
Megnolioniil
THOC
Betala
ACRU
389
5174
363
4423
1136
2257
3905
2102
19474
64755
Val
6x2
68
169
1561
2570
129
12
125
952
465
81
350
3346
689
673
3632
1161
17343
Text
800
76
1964
4632
242
92
583
1083
9934
129
5125
25023
Table 9: Number of instances per class per split in the SBL dataset. Note that the number of instances
is higher than the number of trees since we have overlapping tiles.
B
Models
B.1
SAM automatic
We provide more examples of predictions of SAM in its automatic mode on the Quebec Plantations dataset in Figure 5.
B.2 SAM+DSM prompts
In Figure 6, we show an overview of the SAM+DSM prompts method described in Section 4. Details on how local maxima obtained provided in Appendix B.6. Figure shows some examples of local maxima that are fed as prompts to the mask decoder. One limitation of this method is in the case where there prompts that do not correspond to a tree (third row of the Quehec Plantations column). The case of SBL shows that manual tuning of single neighborhood size parameter to define the height prompts has its limitations. While the chosen parameter is suited for areas with smaller trees (rows and 2). it leads many prompts on the same object for larger trees. The many prompts on the BCI dataset images are due to the fact that the DSM is only given at I m-height resolution, so clustered points often correspond to points with the same DSM value. Note that having too many DSM prompts on tree crowns even if they are not in the center is not major limitation of this method, aside from computation time. Indeed, each point fed independently to SAM, and we apply NMS the predictions, so if segmented objects overlap, only the object with highest confidence score is kept. While there are many ways manually refine nules for filtering local maxima, we did not do any filtering. We originally experimented with setting thresholds on the DSM or tuning further the size of
lot of small plants sticking out of the ground, giving many local maxima
18

---

300
Number of instances
250
100
50
Molvsccaaa
Fabacnee
Bignonaacess
Rubiccaee
Lauraceed
Urticacnoe
Burseraceed
Mellacone
Sapaiacces
Euphortiacree
Apacynaseas
Synisccacee
Ratacees
Maraceae
Arecacese
Cordiaceao
Aracard pncardiacene
Othe
Combreteccos
Noctaginaceoo
Simaroutaaced
Annonacesa
Phyllanthaceed
Lecythidaccon
cenmelaredd
Putranjua Autranivaaesse
Chrytobalenaee eesabeeenceeee
Elaenca Claencarpacen paccop
Salcaccos
Calppity eelottyllccoee
Sopindiceae
Vielacean
rolygenacese
Myytaceen
Staphyleacced ieeee
Araliacene
Rhirophoracee Murophoracees
Clusiacnas
Vochysiaceas
Lythraceao
Enythrepaincenn
Eyethragylareed arcee
Selonacsed
Monimascese
Malnighacesn
Melestorattin
Family label
Figure 4: Distribution of trees of each of the considered families in the BCI dataset, ordered by decreasing prevalence.
the neighbourhood to define local maxima. However, there is tall herbaceous vegetation in some sites, and well-tuned parameters on site would not necessarily transfer well other sites within the same dataset. Also, the DSM does not provide height with respect to the ground but rather relative height between objects the image, and does account for differences in terrain elevation. While it could be possible to normalize the DSM with respect to the lowest point in site, if we have imagery from a site on very angled slope, filtering local maxima based on threshold would not necessarily bring much improvement.
B.3 SAM+DSM mask prompts
We also tried feeding a normalized DSM to SAM as a mask prompt. SAM normally calls for binary mask prompts, and feeding the DSM mask prompt would give gridded segmentations which were not satisfactory enough to be included this study, as shown in Figure 8.
B.4 Mask R-CNN+SAM
Figure shows examples of predictions Mask R-CNN and Mask R-CNN+SAM (in which boxes and masks of the former fed to SAM). While SAM can refine Mask R-CNN segmentations successfully in some cases (see first row), it also leads to gridded segmentation patterns, derading the segmentations overall (second and third row).
B.5 BaISAM variations
We also propose variations on BaISAM, using the addition of the image embedding and the output of the DSM encoder input to the prompt encoder. We show overviews these variations in Figure 10.
19

---

Image
(
(b)
(e)
(6)
(d)
Figure Examples of SAM automatic predictions: (a) A success case. (b) SAM segments everything, including the background, and merging two touching crowns into single instance in the top right corner. (c) SAM segments only the background, i.c., everything but the objects of interest. (d) A lot of tiny isolated objects segmented. (e) SAM completely misses the objects of interest. (f) SAM segments the trees and also the large bushes around.
image embeedinng
Image
image encoder
mask decoder
DSM
Predictions
get local
Feed as point
prompts
frozen
Figure 6: Overview of our SAM+DSM prompts method.
B.6 Implementation details
We first evaluate SAM in its automatic mode on the test set tiles with a points per side (pps) value of 100 (default parameter) and 10. For SAM+DSM prompts, the local maxima in the DSM obtained with skimage. feature. peauureeeeaa...... soak_local__ _eccl_max max function, setting the parameter for minimal allowed distance separating peaks 50 for the Quebec Plantations dataset and 20 for the SBL and BCI datasets. We also tried using scipy. ndimage mwa.immm..mmmmmmmm. ficiiimmmmllll filter to find the local maxima but this led poorer performance. For all SAM out-of-the-box methods, NMS is applied the predictions with a score threshold of 0.5 and overlap loU threshold of 0.5.
All Mask R- R-CNN-based models the torchvision implementation of Mask R-CNN and trained with SGD optimizer with learning rate 0.0001, momentum 0.9, and weight decay 0.0005, and linear warmup starting at 10 Models trained for maximum of 100, 300, and 500 epochs the Quebec Plantations, SBL and BCI datasets respectively. Batch size 32 for Mask R-CNN and for Mask R-CNN+DSM NMS is applied with the default parameters. We initialize the ResNet-50 backbone of Mask R-CNN+ DSM with ImageNet weights, and for the first layer, copy the weights to the channels corresponding the RGB input.
All Faster R-CNN-base models use the torchvision implementation of Faster R-CNN are trained for maximum of 100 epochs on the Quebec Plantations dataset, and Adam optimizer with learning
20

---

Quebec plantations
SBL
BCI
Figure 7: Examples of images with overlayed local maxima prompts for the Quebec Plantations (left column), SBL (middle column) and BCI (right column) datasets.
App
Pradecto
Figure 8: Examples of image and prediction when the DSM is fed mask prompt to SAM.
rate 0.0001 for finetuning and 0.0005 when trained from scratch, betas of 0.9 and 0.999, weight decay of 0.0005, and using an exponential decay scheduler updating the learning rate each 10 epochs. Batch size is 32 for Faster R-CNN and 16 for Faster R R-CNN+DSM, NMS is applied with the default parameters. We initialize the ResNet-50 backbone of Faster R-CNN-DSA DSM with ImageNet weights, and for the first layer, copy the weights to the channels corresponding to the RGB input.
For Faster R-CNN+SAM and Mask R-CNN+SAM methods, the scores used to compute the mAP metrics are the average of the output scores of Faster R-CNN/Mask R-CNN and SAM predicted loU scores.
For the Farkerormer models, we used the Swin-base and the Swin-large versions the model, and initialized the models with COCO coCO-pretraieed weights, using the implementation available through Transformers. The default preprocessing and postprocessing for evaluation were left unchanged. Batch size is 8 for the Swin-base and 4 for Swin-large large Fask2Former models. Swin- base models trained for maximum 100 epochs, Swin-large models are trained for a maximum of 50 epochs, datasets.
these models tend to overfit
Following Chen et [20], the RSPrompter based methods are trained with input images of size x 1024, normalized with ImageNet statistics, and learning rate scheduler strategy of linear
21

---

Mask S-CN
Figure 9: Examples of ground truth, Mask R-CNN and Mask R-CNN+-SAM predictions the Quebec Plantations dataset.
imagee emoedding
peompt encoder
Teage
Image wudoobling
SAN
SAM mud: decodar
you encoder
encoder
SAM imapp umander
SAM met denaker
DSM
DSM encnder
Prasioives
frorr
(b) In this variation, the prompt encoder receives the added image embedding and DSM emhedding, and no prompt is fed in the dense prompt branch mask decoder.
tranabli
(a) variation, prompt encoder receives the added image embedding and DSM embedding.
Figure 10: Variations on BalSAM
warmup followed by cosine annealing. The models are trained with batch size 2 (as in Chen al. [20]'s experiments), base learning rate of 0.00001 with linear warmup starting at 10 for one epoch followed by cosine annealing. We use AdamW optimizer with weight decay 0.1. Models are trained for maximum of 50, 100 and 200 epochs on the Quebec Plantations, SBL. and BCI datasets respectively. The DSM encoder of BaISAM is 3-layer CNN with layer normalizations and GeLU activations. The CNN layers are defined as following:
First layer: Kernel size 2), with 192 output channels, and a stride of (2,
Second layer: Kernel size (8, 8), with 768 output channels, and a stride of (8, 8).
Third layer: Kernel size (1, 1), with 256 output channels, and stride of (1, 1).
All the models were trained on a single RTX8000 or A100 GPU, requiring up 48GB GPU memory and 24GB CPU memory. Only the Mask DSCNN+DSM encoder model and variations on BaISAM, presented in the Ablations required a larger GPU, and were trained on a A100 GPU with 80GB GPU memory and 48GB CPU memory. Experiments took between one and five days to complete, depending models and batch size.
22

---

While RSPrompter and BalSAM use batch size of 2 due to the large input image size of 1024x 1024 pixels, following [20], and the models were left train until the maximum number of epochs was reached, the best model (selected with the validation set mAP) is usually trained for fewer epochs than Mask R-CNN-basee models.
We report average inference speed per sample for different models using the same V100-SXM2-32GGB GPU in Table 10. While did not control for the type of GPU used across experiments and therefore training cost figures would be misleading, we report the number of trainable parameters in each model (this includes the final layer for the Plantations dataset, which has 9 classes). For models that can also take the DSM as 4th channel along the RGB images input, the addition of the DSM does not lead significant increase in number of parameters.
Model
Inference speed (s/image)
Number of trainable parameters
SAM automatic
15.3
0
SAM+ height prompts
1.97
0
Mask R-CNN
0.34
43.7M
Mask2Former Swin-base"
0.27
106.9M
Mask2Former Swin-large"
0.30
215.5M
RSPrompter
1.00
1 117M
BalSAM
1.04
126M
Table 10: Comparison of average inference speed on the Plantations test dataset and number of trainable parameters of different models ("note that Mask2Former models used images that are 384x384 pixels inputs when all the other models used 1024x1024)
C Results
C.1 Class-wise performance on the Quebec Plantations dataset
Figure 11 shows the per-class mAP performance of different models on the Quebec Plantations test set.
0.5
0.7
0.6
0.5
0.4
Model
Rbok C.N
Max KCW DOMN
0.0
pial
thoc
other
pba pima Class
u'm
acca
boal
Figure 11: Per mAP performance on the Quebec Plantations test set. each model, the performance averaged on seeds and we show standard deviations. Tree species on the x-axis are ordered by decrcasing prevalence in the dataset from left to right. Mask R-CNN is pre-trained on ImageNct. Numerical results are provided in Table 11.
In Table 11, report the per class mAP on the test set for the different methods in our study, the exception of the SAM out-of-the-Be methods which do not classify the predicted masks. We also show confusion matrix for predictions of Mask R-CNN model on the Quebec Plantations dataset in Figure 12.
C.2 Class-wise performance on the SBL dataset
We report per-class mAP on each the classes in the SBL. test set in Tables 12 and 13. We order classes in the tables by decreasing prevalence in the training set. We observe that RSPrompter and
23

---

Model
DSM
pre-trained
piba
pima
pist
pigl
thoc
ulam
other
beal
BCSa
Mask R-CNN
K
45.88
13.31
73,60
69.75
60.08
30.90
41.82 20.12
7.64 16.30
41.23 1367
Mask R-CNN
X
51.55 1:029
10.59
74.67 10:20
73.14 10002
61.53 10:77
35,49 11.15
45.94
10:28
9.81 15.08
$5.78 11.58
Mask R-CNN
/
/
53.08 10.10
21.56 1368
76,97
10.45
73.29 10.29
62.57 1061
38,30 10.3
46,43
10.40
10.78
1:42
57. $7,69 10.76
Faster CANN
X
60.56 1005
56.28
10.51
28.96
10.55
24.53
10.1
3.32 10:31
26.26 21.33
20.4
41.2 13:7
Faster R-CNN+SAM
X
V
64.11
61.03
34,93
31.57
3.61
33.58
66.65
49.82
12.84
Faster R-CNN+SAM
V
V
66,04 10.6
61.38 10277
35,46
10.06
30.78
10.18
17.26 17256
31.94
10.39
66.37
10.35
51.86 10.57
0.15 10.15
RSPrompter
X
53.03 1:029
29.17
19.53
76.83
1:047
73.23 1:.45
69.43 1.0.5
44,40
10.12
47.33 #0.37
16,00 #7.99
65.43 #:265
BalSAM
51.17 1009
50.97 16.15
75.07 10.35
76.47
1.011
67.93 10.57
43.07
11.12
46.87 10.34
13,93 19:32
64.23 15.3
Table 11: mAP per class [102] with standard errors on the Quebec Plantations test set for the instance segmentation models in our study.
Confusion Matrix
3462
0
0
0
6000
527
0
0
0
4000
Thus Label
106
25
100
56
1356
104
23
0
3
3000
19
other
18
15
0
5
0
1000
-9
pbba pima
pist
pig! thoc ulam
beal acia
Predicted Labrl
Figure 12:
Confusion matrix for the Mask R-CNN model predictions on the Quebec Plantations test set. Picea mariana (pima) is most often confused with Picea glauca (pigl), which is very similar looking species, and the most common the dataset.
BaISAM significantly improve on Mask R-CNN methods on less common classes in the dataset, in particular for Picea, Fagus grandifolia (FAGR) and Tsuga canadensis (TSCA) classes.
Model
BEPA
ACRU
ABBA
Populus
ACSA
THOC
Dead
ACPE
PIST
Mask R-CNN
33.87 # 0.19
21.84 +: 0.12
35,59 + 0.13
40.47 10.0
19.64
10.48
29.65 10.6
18,30 + 0.66
9.93 a 0.72
44.63 + 0.14
Mask R- CNN+DSM
33.61 0.02
21.74 +: 0.25
35.31 039
41.55 = 0.88
17.99 027
28.68 + 0.45
17.85 0.03
9.72 + 0.3
44,90 +: 1.48
RSPrompter
34.98
23.08 + 0.64
36.48
1.10
43.77 = 0.50
20.92
33.14 1 0.47
22.63
088
10.90 + 0.64
48.12
BaISAM
34.76 1.02
22.12 = 0.75
36.53
= 0.43
45.73 # 1.52
20.96 1.05
32.58 = 073
21.38 1.71
10.46 + 0.25
47.53+ 1.46
Table 12: Per-class mAP on the SBI. dataset for the most prevalent classes in the training set (ordered from left right in decreasing order of prevalence)
Model
Picca
Accr
LALA
Magnoliopsida
FAGR.
BEAL
TSCA
Pinopsida
Mask R-CNN
31.93 10.5
0.00 + 0.01
40.23 +:2 1.09
0.00 10.00
15.09 + 0.99
21.66 20.35
0.53 + 043
0.00 20.00
Mask R-CNN+DSM
30.92 20.8
0.00 20.00
40.76
+1.09
0.00 10.00
11.68 10.15
20. 1 +1.39
0.00 10.00
0.00 10.00
RSPrompter
40.83 =0.50
1.01
#0.30
43.92
20.79
0.86 #0.43
18.17
11.65
21.67 11.29
23.50
0.02 #0.01
BaISAM
40.73 #0.9
1.11
=0.05
45.13
#1.15
0.25 #0.07
17.69 10.09
22.62 40.93
23.38 #3368
0.00 10.00
Table 13: Per-class mAP on the SBL dataset for the least prevalent classes in the training (ordered from left right in decreasing order of prevalence)
D Ablations
D.1 Informativeness of the DSM
We trained a Mask R-CNN with DSM inputs only (no RGB image) to assess its informativeness on
the Plantations dataset vs the SBL dataset and report results in Table 14. When looking at the drop in performance compared to the Mask R-CNN models using RGB images inputs, we observe
24

---

much larger drop on the SBL dataset compared to the Plantations dataset. In fact, single- mloU remains high, which another way to see that tree crowns are distinguishable from the background and from each other using the DSM only, and points to the usefulness of the DSM in plantation contexts.
Dataset
single-class mAP
mloU
mAP
Plantations
0.464
0.734
0.178
SBL
0.092
0.289
0.023
Table 14: Test set performance of Mask-R-CNN (pre-trained on ImageNet) using DSM only input
D.2 Mask R-CNN+SAM prompts and scores
We explore using mask predictions, box predictions or both, output by a trained Mask R-CNN, as prompts to SAM. Additionally consider different prediction scores, using either box scores only from the Mask R-CNN, or the average the IoU scores of SAM and the box scores of the Mask R-CNN for computing the evaluation metrics. We report performance the SBL dataset for different combinations of scores and prompts in Table 15.
Single-class
Multi-class
hox prompts
mask procppts
box score
maske box score
mAP
mloU
mAP
WmAP
24.59
Mask R-CNNASAM
/
20.46 20.14
26.21 #o.17
58.59 20.30
61.67 20.35
14.73 20
18.23 20.17
17.24 20.1d
21.83 =0.00
/
22.95
58.58
16.06
19:00
7
Mask R.CNN-SAM-DSM
22,83
20.47 10.18
10.30
58.15
58.16
20.20
14.49 20 11
15.77
17.05 10
18.71 g000
Table 15: Comparison of using different mask and box prompts and scores for the Mask R-CNN+SAM-base models on the SBL test set. highlight the best combination prompts and scores bold for Mask R-CNN+SAM and Mask R-CNN+SAM+DSM.
D.3 Incorporating DSM information
We report results for different Mask R-CNN-based models incorporating DSM information on the SBL test in Table 16.
To obtain DSM gradients, we used numpy. numpy-gradient gradient with spacing to get vertical and horizontal gradient maps. Some tiles the border of AOIs have black pixels, which would lead to very high gradient values between the black areas and the image area. In this case, we paste mask of of zeros, covering to the black pixels area, onto the DSM gradient maps.
For the model with extra capacity in the Faster R-CNN head of Mask R-CNN, we added an extra I. Linear layer followed by ReL.U activation before the output layers of the bounding box predictor and the classifier of Faster R-CNN.
For the model with an added DSM encoder, the DSM encoder architecture is 3-layer CNN with layer normalizations and GeLU activations. The CNN layers are defined following:
First layer: Kernel size 2, with 192 output channels, and same padding.
Second layer:
Kernel size (2, 2), with 768 output channels, and same padding.
Third layer: Kernel size (1, with output channel.
The output is the same size the original DSM. This setup was used on the SBL and BCI datasets. Note that the Mask R-CNN+DSM encoder models were trained on single GPU with 80G GPU memory and 48G CPU memory.
25

---

Single-class
Multi-class
Model
mAP
mloU
mAP
wmAP
DSM
32.37 +0.18
64.08 #0.17
20.87 #0.13
26.82 #0.15
DSM gradients
32.43 1:041
64.55
1:033
20.68
10.17
26.91
1:029
Extra capacity in Faster R- R-CNN head
20.26
64.55
10.35
20.82
10.08
26.95
10.16
DSM encoder
32.35 10.17
64.80
#0.20
20.54 +0.20
26.76 10.16
Table 16: Results for different Mask R- CNN-based models incorporating DSM information on the SBL set. Metrics are multiplied by 102 and reported with standard error.. We highlight the best model for cach metric in bold.
D.4 Losses
D.4.1 Hierarchical loss
We define hierarchical loss, modifying it from [63] since consider different classes of interest. Similarly to [63], we consider 3 losses, "species", "genus" and "family" the species loss, exclude instances that have ground truth labels in [Betula, Acer, Magnolop- sida,Pinopsida). In other words, only instances that have a species- level label or that do not have any subcategory the species-level in the annotations contribute to the loss. For example. we include Picea as class contributing to the species loss, because there is no class that corresponds to a finer Picea species-level. When computing the genus level loss, exclude instances that have ground truth labels in Magnolopsidd, Pinoosidaa We use the same weights as [63] for species, genus and family losses in the final loss.
When computing
D.5 Results for different losses
We summarize results for Mask R-CNN models trained with different losses in Table 17. Hierarchical loss improves slightly but not significantly the cross-entropy used in all our experiments.
Single-class
Multi-class
Loss
mAP
mloU
mAP
wmAP
Cross-entropy
32.44 10.12
65.08 #0.41
21.38 #0.17
27.27 #0.18
Weighted loss
13.23 10.17
62.05
10.17
8.10 10.18
12.32.10.31
Hierarchical loss
32.76 100.66
64.70
10.25
21.42 10.15
27.51 10010
Focal loss
30.79 +0.39
64.55
+0.25
14.63 +0.21
23.49 =0030
Table 17: Results for different Mask R-CNN the SBL test using different losses. We highlight the best model for each metric in bold.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We highlight contributions in the abstract and introduction. This is the first benchmark of instance segmentation methods and assessment of the difficulty of this task on the three considered datasets. Experimental results support the claims that integrating DSM information into models can improve predictions and that methods learning prompt SAM end-to-end are advantageous over SAM cut-of-the-bor and Mask R R-CNN-base -bssed models.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of this work are highlighted in the conclusion section of the paper. We also discuss computational efficiency in Appendix B.6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
27

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Processing details of the dataset, implementation details of the models and choices of hyperparameters are provided in Sections 3 and 4 of the paper and in Appendices A and B.6.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
28

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We existing, publicly available datasets, described in Section 3. Data preparation steps details are provided the paper, as well as training details to reproduce the experiments. Code pre-process the open-source datasets used in this study and train different methods presented in this paper is available at meli sandeteng/ BalSAM.
com/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines fattps://nips. cc/
public/guides http:///oeeuusiiiinneeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Details on training and test details are provided Section 4 and Appendix B.6. We also share the AOIs used to define the splits in the Supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Standard error or the mean is provided for all reported metrics for all models.
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer:
[Yes]
Justification: We specify compute requirements for our experiments in Section and Appendix B.6.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:: ///errips//////...//// hwtpeeewwwwsp.pppppeeeeeeeeeeeee hetpuliie/Ehicsuuddeeee ines?
Answer: [Yes]
Justification: This work does not involve human subjects participants. The data used is open-source and terms of their licenses have been respected. Societal impact and potential harmful consequences outlined in the conclusion section of the paper. Models and code will be released publicly responsibly upon paper decision.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
30

---

Answer: [Yes]
Justification: Both potential positive and negative societal impacts of the work are discussed in the conclusion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pre-trained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not releases new data and the models presented in the study do not have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Creators of assets on which the paper is built on (code, data and models) are properly credited in the paper, and their licenses and terms of use are properly respected. We share code
Part of the code is based a fork of RSPrompter/20]]s original implementation, available at https: / githhu..
com/mel /memmmleeeee www.ldittengaaaalaaaa.
31

---

Nom/KyanChen, http/yyaanchee//PPPoptttrrr /SSPooppt..rrr.. Names of licenses for existing assets are also explicitly mentioned in the code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/coo/////oo/oo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide representative sample the code in the Supplementary material with supporting documentation. A full version of the code with further documentation will be released upon paper decision.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not include crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
32

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This work does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy (https:// http://rpps.ccmmmmeeeeeeeeeeeeee for what should or should not be described.
33


Reviews:
[{"review_id": "cM1sgLRkgI", "reviewer": "Reviewer_6LSe", "summary": "This paper investigates the application of the Segment Anything Model (SAM) for tree crown instance segmentation from high-resolution drone imagery. The authors introduce BalSAM, which utilizes the RSPrompter framework to incorporate Digital Surface Model (DSM) data for improved segmentation. The study benchmarks several methods—including out-of-the-box SAM, prompted SAM, Mask R-CNN, and RSPrompter—across three distinct forest environments: boreal plantations, temperate forests, and tropical forests. The findings suggest that while off-the-shelf SAM performs poorly, methods that learn to prompt SAM (RSPrompter, BalSAM) marginally outperform a conventional Mask R-CNN baseline. The integration of DSM data is shown to be marginally beneficial, particularly in the structured context of plantations.", "questions": "1. The core innovation of BalSAM is the addition of a DSM encoder to RSPrompter. Given that BalSAM does not consistently outperform RSPrompter across all datasets and metrics, how do the authors assess the overall usefulness of DSMs? Why could adding DSM information degrade performance, as seen on the BCI dataset?\n2. Could the authors provide a more direct analysis of the trade-offs between accuracy, inference speed, and training cost? Without this, it is difficult to judge whether the modest mAP improvements justify the substantial increase in computational resources when using SAM and RSPrompter.\n3. Could the authors at least compare with pretrained Mask2Former, which seems to be a fairer baseline than MaskRCNN, given that the model being evaluated is transformer-based SAM?", "limitations": "The authors largely ignored the limitations in the wide applicability of the proposed method due to the limited availability of high-resolution DSM.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "### Strengths\nTree crown delineation problems are well-motivated and significant problems in forest monitoring.\n\n### Weaknesses\n1. I have deep concerns over the significance of the results: The main hypothesis of the work is that DSM can help tree crown segmentation. However, the results presented in Section 4.2 suggest that adding DSM has limited improvement, if not the opposite. For example, in Table 1 (test results on the Quebec Plantations dataset), BalSAM achieved an mAP of 65.03 while RSPrompter achieved 66.37. Similarly, in Table 2 (multi-class), RSPrompter is better than the proposed BalSAM.\n2. Mask R-CNN and Faster R-CNN with a ResNet backbone are unfair comparisons to SAM, which is based on ViTs with a much larger number of parameters. I encourage the authors to perform additional evaluations with models like Mask2Former [1].\n3. I also have deep concerns over the originality of this work. As both SAM and RS-Prompter are previous research, this work's contribution is about adding DSM data into the RS-Prompter pipeline, which is not always helpful. To improve the originality of this work, I encourage the authors to include more discussions around the useful signals to perform tree crown segmentation and provide more insights into the failure points of existing methods, including the proposed BalSAM. \n\n[1] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A. and Girdhar, R., 2022. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1290-1299).", "was_revised": true, "final_justification": "After the discussion, my questions regarding the trade-offs between accuracy, inference speed, and training cost have been addressed.\n\nHowever, I'm not sufficiently convinced that using DSM to prompt SAM is an effective way to fuse DSM information into the model, as the evaluation results are so close to, if not even worse than, the baseline. The authors acknowledge that the evaluation numbers may not fully reflect the model's performance, as certain mistakes in crown segmentation can have a large influence on downstream tasks like AGB estimation. The current experiments do not demonstrate such advantages of BalSAM through additional case studies of downstream applications.", "rebuttal": "Dear reviewer, \n\nWe thank you for your time and detailed review, and recognizing the importance of this forest monitoring application.\n \nThe goal of this work is two-fold: investigating the task of tree crown instance segmentation with respect to **(i) whether the DSM helps for certain forest types and architectures**, and **(ii) when and how SAM is useful**. Both questions have not been explored extensively before and our work is the first to compare models across those two axes of analysis, even though SAM is the most readily accessible model to many practitioners and the DSM is a product that is always readily available at no additional cost from the RGB imagery.  \n\nWe show **1/ that using the DSM alongside RGB imagery improves segmentation and classification mean results in the case of tree plantations, which have a specific structure, 2/ in mature forests, the DSM either improves results or keeps them comparable within error bars, 3/ while SAM out-of-the-box does not perform well, SAM-prompt based methods (RSPrompter, BalSAM), which have so far not been used for tree crown segmentation, reach best performance overall on all the datasets.**\n\nOur work is a first step towards guiding practitioners in their choices of models for forest monitoring, where different use cases may call for different methods depending on the nature of the data. \n\nWe would also like to highlight that **what might seem like small average differences in evaluation scores can translate to large differences in downstream applications**. Consider the equations used in [3] to determine the amount of carbon stored in trees of different species of the form y_i = beta1_i* (diameter^beta2_i)*(height^beta3_i) where y_i is the biomass of part i of a tree (i = foliage, bark, …). The total biomass of the tree is the sum of the y_i. If one considers y_foliage, the coefficients (beta1, beta2, beta3) are (2.5, 2.45, -2.3) for sugar maple trees and (0.2, 2.38, -1.11) for white spruce - thus, making classification mistakes in an instance segmentation task can lead to huge differences in carbon estimates, especially for small trees (which is what we have in the Plantations dataset).\n\n**We respond to your questions and concerns below.**\n\n**Weakness1/Q1/Limitation**:  While it is true that BalSAM does not always offer a significant advantage over RSPrompter,  our results show that BalSAM either outperforms or performs similarly (within error bars) to RSPrompter across use cases.  As outlined in the discussion (5.1), a reason for the lesser informativeness of the DSM on the BCI dataset could be the complex structure of canopies in tropical forest in which there is high overlap between tree crowns and species diversity,  than in the Quebec Plantations dataset. \n\nRegarding **assessing the overall usefulness of the DSM**, our study shows that the DSM proves useful in most of the experiments when added to RGB information. When the DSM is not helpful (e.g. Mask R-CNN baseline on the SBL dataset), this seems to be due to the structure of the forest which practitioners can assess (see examples of DSMs in figure 1).  \nWe recommend practitioners to use the DSM along RGB images for plantation use cases where we showed that it consistently improves performance for all methods in terms of the multi-class mAP and wmAP scores.\nWe then leave it to users to decide which method is more suited for other use cases, based on which data they have available. For example, if one only has access to bounding box annotations to train their model, the Faster R-CNN+ SAM method could be used and we showed that adding the DSM in this context could bring significant improvements compared to using RGB only. \n\nRelatedly, we would like to respond to your concern in the “limitations section” about “limited availability of high-resolution DSM”, and clarify that the **DSM is in fact always readily available at no additional cost**  when one acquires drone imagery. Structure-from-motion (Sfm) photogrammetry, which is used to create RGB orthomosaics from high-resolution RGB drone imagery, generates 3D photogrammetry dense point clouds, from which the DSM is derived. The 3D point cloud is actually needed for orthorectification to obtain RGB orthomosaic. In other words, the DSM is a by-product of the photogrammetry process and is thus available at no additional cost from RGB drone data processed with Sfm. This is a central point and a key motivation for our study, which we will further emphasize in the paper.\n\n**Q2. “... analysis of trade-offs between accuracy, inference speed, and training cost.”** \n The table below shows average inference speed per sample for different models using the same V100-SXM2-32GB GPU. While we did not control for the type of GPU used across experiments and therefore training cost figures would be misleading, we  report the number of trainable parameters in each model (this includes the final layer for the Plantations dataset, which has 9 classes). For models that can also take the DSM as a 4th channel along the RGB images as input, the addition of the DSM does not lead to a significant increase in number of parameters.\n| **Model**   | **inference speed** (s/image)| **Number of trainable parameters**\n| --- | ---|---|\n| SAM automatic | 15.3 | 0 |\n| SAM+ height prompts | 1.97 | 0 |\n| Mask R-CNN  | 0.34|43.7M  |\n| RSPrompter | 1.00  |   117M | \n|  BalSAM   | 1.04 |  126M |\n\nWe would like to note that during inference time in real world scenarios, we do not have significant constraints (apart from using a single GPU) on the size of the models and their inference speed since the operation is mode offline. Typically, organisations looking into assessing the carbon stored in their plantations do not need embedded system models. \n\n**Weakness2/Q3**. We thank the reviewer for raising the valid concern of comparing models with different capacities and suggesting to add experiments using more recent ViT-based models. **We ran Mask2Former with Swin-base and Swin-L backbones on the Quebec Plantations dataset** using RGB or RGB+DSM as input; we report results in the table below. The models have 106.9M and 215.5M parameters, respectively. They were initialized with COCO-pre-trained weights, trained for a maximum of 100 epochs (as were the other models presented in the paper) and took ~1.5 days to train on a single GPU. Given the timeline of the rebuttal, we performed only minimal HP tuning on the learning rate and learning rate scheduler. We find that the DSM improves performance for Mask2Former Swin-base. It is possible that results could be further improved and that Swin-L models require longer training. We are currently conducting further HP tuning with Mask2Former and will add results on multiple seeds in the main paper. That being said, several previous works have highlighted limitations of Mask2Former in the context of forest monitoring from remote sensing imagery. \nFor example, in [1],  the authors highlight: “There are several transformer-based successors to Mask-RCNN, from the DETRfamily of models to Mask2Former. While we experimented with these architectures, [...], we were unable to reliably outperform Mask-RCNN and found poorer convergence behaviour”. [2] also find that  Mask2Former models’ “performance is not comparable to the CNN-based architectures” in the context of semantic segmentation on the SBL dataset. \n\n| **Model**   | **mAP single-class** | **mIoU single-class** | **mAP** | **wmAP**\n| ---| ---|---|---|---|\n| Mask2Former Swin-base RGB| 55.2|70.0|34.3|37.5|\n| Mask2Former Swin-base RGB+DSM| 58.1|71.9|37.1|39.6|\n| Mask2Former Swin-L RGB| 62.7|74.7|45.1|46.0|\n|Mask2Former Swin-L RGB+DSM|62.1|73.8|42.0|44.8|\n\n**Weakness3: “...encourage the authors to include more discussions around the useful signals to perform tree crown segmentation and provide more insights into the failure points of existing methods”** \nThank you for your helpful feedback. We will expand on the discussion and future work to cover these topics more comprehensively, as follows:\n\nThe usefulness of the DSM is (unsurprisingly) linked with the structure of the canopy. Models also tend to perform less well in contexts in which we have overlapping crowns.  As an ablation, we ran a **Mask R-CNN with only the DSM** as input (no RGB imagery) on the Plantations and SBL datasets and report results below. We see a much larger drop in mIoU on the SBL dataset when comparing to Mask R-CNN models using image inputs, while on the Plantations dataset, mIoU remains high, which highlights the informativeness of the DSM in the plantations case. \n\n| **Dataset**   | **mAP single class** | **mIoU** | **mAP** \n| ---- | ----|----|----|\n| Plantations  | 46.4 |73.4 | 17.8\n| SBL | 9.2 |28.9 |2.3\n\nWe also believe that a promising avenue for future work is adding contextual metadata into our models in the form of spatial information, spectral information, and the type of forest, to improve the performance of tree crown instance segmentation models. While we have so far trained models separately on each dataset, as more high-resolution drone imagery data becomes available, it could be very interesting from an application perspective to learn representations on combined datasets at different resolutions. Location metadata could also prove helpful, as it has been shown to in species distribution modelling contexts. Spectral information, available through e.g. satellite open-source programs, could also be added as additional input signal into the models even if it is not available at a resolution as high as that of the drone imagery.\n\nWe hope this addresses your concerns and we are happy to answer any further questions you may have. \n\n[1] _Veitch-Michaelis, J., et al. \"OAM-TCD: A globally diverse dataset of high-resolution tree cover maps.\"(2024)_\n\n[2] _Ramesh, V., et al. \"Tree semantic segmentation from aerial image time series.\" (2024)_\n\n[3] _Lambert, M.-C., et al. \"Canadian national tree aboveground biomass equations.\" (2005)_"}, {"review_id": "Qk9abui0wd", "reviewer": "Reviewer_XmBk", "summary": "Accurate individual tree crown detection is a essential component of precision forestry. This work proposes BalSAM, which integrates Segment Anything Model with the DSM elevation information for instance segmentation and species classification of individual tree crowns in UAV images. The paper systematically evaluates the method on three typical forest datasets. The results show that: 1) The out-of-the-box SAM has limited effect; 2) Fine-tuning SAM through prompt learning and introducing DSM can achieve mAP improvements; 3) In dense forests with many species, DSM still brings some segmentation benefits but has little impact on classification.", "questions": "Q1: Explanation on the low accuracy of multi-class classification.\nQ2: Discussion on latest Transformer-based instance segmentation or fine-tuned SAM.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S1. This study is the first to apply SAM to the tree crown instance segmentation task, and integrated DSM height data to improve the segmentation. The proposed BalSAM draws on the segmentation ability of SAM, while achieving cross-modal data fusion by adding height information, and expands the application of SAM in remote sensing.\nS2. It covers three representative forest datasets with significant differences, including artificially planted forests, temperate forests with diverse species, and tropical rainforests with high canopy density, and compares multiple baselines such as Mask R-CNN, Faster R-CNN + SAM, etc. The conclusion is representative.\nS3. Integrating DSM information has improved the segmentation and classification accuracy of the model, especially in planted forests where tree distribution is regular and canopy separation is distinct. The tree crown contours segmented by BalSAM are more in line with the actual shapes and the model effectively reduces the occurrence of species misclassification.\nS4. Digital Surface Model information is derived from RGB drone imagery. It does not require additional sensors and has direct application potential for resource-constrained forestry monitor.\nW1. On datasets with multiple species such as tropical rainforests, the multi-class mAP remains at a low level (≤ 9%), and its contribution to fine-grained classification is limited.\nW2. The article lacks a comparison with the latest Transformer-based instance segmentation or fine-tuned SAM, and the superiority argument is somewhat insufficient.\nW3. The standard error of the result is relatively large, further robustification might be necessary in actual deployment.", "was_revised": true, "final_justification": "After reading the authors' response and other reviewers' comments, my previous concerns have been well addressed and I do not have further concerns on this paper. I will raise my score to \"Accept\"", "rebuttal": "Dear reviewer, \n\nThank you for your helpful review and for valuing our contribution to the field of forest monitoring with this first study on applying SAM and integrating DSM information for tree crown instance segmentation. \n\nWe respond to your questions below. \n\n**W1/Q1** Indeed, the classification accuracy remains low overall on the BCI dataset. As we briefly mention in the discussion, we think this is due to the higher diversity of trees in this tropical context and to the lower resolution and quality of the images from this dataset compared to the other datasets. Additionally, annotations of the BCI dataset are noisy (as can be seen in Figures 1 and 3) and some tree crowns are not annotated. This reflects the complexity of the task of delineating their crowns and classifying them, even for experts in biology. The examples in Figure 3 (rows 3 and 4) show that our models sometimes segment trees that were not labeled. This may also lead to an underestimation of the performance of the methods. \nA potential avenue for tackling this challenge is to continue the work on using different losses that we present in D.3, as well as to explore more losses suited for long-tailed distributions.\n\n**W2/Q2** Upon your suggestion, **we conducted experiments with Mask2former using the Swin-base and Swin-L backbones on the Quebec Plantations dataset** using RGB or RGB+DSM as input. We report results in the table below. \nThe models were initialized with COCO-pre-trained weights, trained for a maximum of 100 epochs (as were the other models presented in the paper), with batch size 8 and took ~1.5 days to train on a single GPU. Given the timeline of the rebuttal,  we performed only minimal HP tuning on the learning rate and learning rate scheduler. We find that the DSM improves performance for the Mask2Former Swin-base model. It is possible that results could be further improved, and that models using the larger Swin-L backbone require longer training. Therefore, we are currently conducting further HP tuning with Mask2Former and will add results on multiple seeds in the main paper. That being said, several previous works have highlighted limitations of Mask2Former in the context of forest monitoring from remote sensing imagery. \n\nFor example, in [1],  the authors highlight:  _“There are several transformer-based successors to Mask-RCNN, from the DETR family of models to Mask2Former. While we experimented with these architectures, [...], we were unable to reliably outperform Mask-RCNN and found poorer convergence behaviour”_.\n[2] also find that  Mask2Former models’ _“performance is not comparable to the CNN-based architectures”_ in the context of semantic segmentation on the SBL dataset. \n\n| **Model**   | **mAP single-class** | **mIoU single-class** | **mAP** | **wmAP**\n| ---| ---|---|---|---|\n| Mask2Former Swin-base RGB| 55.2|70.0|34.3|37.5|\n| Mask2Former Swin-base RGB+DSM| 58.1|71.9|37.1|39.6|\n| Mask2Former Swin-L RGB| 62.7|74.7|45.1|46.0|\n|Mask2Former Swin-L RGB+DSM|62.1|73.8|42.0|44.8|\n\nRegarding the suggestion to fine-tune SAM, this would unfortunately not solve the problem of instance segmentation - in which we also care about classifying the segmented instances, since SAM predicts masks without classes. Moreover, SAM’s image encoder has 632M parameters and fine-tuning SAM would require considerable compute resources, rendering its training process inaccessible to most forest-monitoring practitioners. In order to train RSPrompter and BalSAM on a single GPU, we used a batch size of 2, while fine-tuning SAM with limited computational resources would simply not be feasible.\n\n**W3**: “The standard error of the result is relatively large, further robustification might be necessary in actual deployment.” We acknowledge that this is a limitation of RSPrompter and BalSAM, as we wrote in L360-361, and agree that further robustification is a priority for future work. This could be done by stabilising the training process through regularization and augmentations, especially on hard classes. We will also further analyse the errors and success cases in the best and worst runs from different seeds of the same model to better understand where issues arise and inform the choice of regularization and augmentations. \n\nWe hope that this answers your questions, and please let us know if you need any further clarifications. \n\n\n[1] _Veitch-Michaelis, J., et al. \"OAM-TCD: A globally diverse dataset of high-resolution tree cover maps.\" Advances in neural information processing systems 37 (2024)_\n\n[2] _Ramesh, V., et al. \"Tree semantic segmentation from aerial image time series.\" (2024)_"}, {"review_id": "w5qBkVzGPu", "reviewer": "Reviewer_XSSG", "summary": "This paper highlights the challenge of identifying and segmenting individual trees from high resolution imagery across different geographic contexts. To address this challenge, this paper presents an approach, BalSAM, to incorporate elevation data in the form of digital surface models into SAM-based workflows to segment individual trees in UAV imagery. This approach is compared to a range of other segmentation approaches including applying SAM to just UAV images and CNN-based tree segmentation models. The authors demonstrate that their approach, BalSAM, outperforms a regular SAM-imagery based segmentation approaches and custom-trained CNN models.", "questions": "Fig. 7 presents examples of prompts generate by local maxima in DSMs, many of these point prompts do not fall on trees. I was wondering if the author’s filtered out points that are obviously not on / in the centre of tree canopies using sensible height thresholds? \n\nThe ablation studies section could be summarised and point to relevant sections in the appendix and a suggestion would be to explaining / discussing challenging cases (e.g. BCI data) and outlining avenues for future research that flow from this work.\n\nWhat cases are causing error in the model for the different datasets? Is it small trees? Is it resolving the shapes of complex crowns / overlapping trees? It would be good to include some information on this in the results / discussion sections.\n\nWhat are the author’s thoughts on how application-ready the BalSAM method is for the use cases outlined in the introduction that need individual tree segmentations with species labels? It would be good to include comments on this in the conclusion section.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The motivation for the paper was clearly articulated -  the challenge of segmenting and classifying individual trees from UAV imagery and the need for this information. The authors have done an excellent job clearly presenting this work; it is well written and the easy to follow the methods and results. There is a thorough comparison of the main approach developed here for integrating DSM’s into RGB image-based segmentation of individual trees, BalSAM, with a range of different models. It was good that the authors evaluated the different individual tree segmentation approaches across a range of geographic contexts. \n\nThe discussion and ablation sections were quite general. There was an opportunity to provide explanation of the results and findings presented in this paper, to discuss considerations for broader application of the BalSAM approach, and to suggest avenues for further research. The authors note the challenges of segmenting individual trees in the dense tropical forest case. It would have been good to hear the author’s thoughts on how to move forward in these cases and to possibly consider opportunities offered by different classification / modelling approaches or ways of summarising information in these scenes for downstream tasks (e.g. estimating carbon or species abundance within larger quadrats / grid cells). It would have been good for the authors to offer suggestions on how to generalise the BalSAM approach to locations where labels are not available for fine tuning.", "was_revised": true, "final_justification": "I stand by my initial recommendation that this paper be accepted.", "rebuttal": "Dear reviewer, \n\n\nThank you very much for your insightful comments and valuing the importance of this application in forest monitoring and the potential for impact of our work. \n\nWe respond to your questions below. \n\n**Q1–Local maxima prompts:**  While there are indeed many ways to manually refine rules for filtering local maxima, we did not do any filtering. We originally experimented with setting thresholds on the DSM or tuning further the size of the neighbourhood to define local maxima. However, there is tall herbaceous vegetation in some sites, and well-tuned parameters on a site would not necessarily transfer well to other sites within the same dataset. Also, the DSM does not provide height with respect to the ground but rather relative height between objects in the image, and does not account for differences in terrain elevation. While it could be possible to normalize the DSM with respect to the lowest point in a site, if we have imagery from a site on a very angled slope, filtering local maxima based on a threshold would not necessarily bring much improvement. Moreover, having too many DSM prompts on tree crowns even if they are not in the center is not the main limitation of this method because each point is fed independently to SAM. If segmented objects overlap, we use NMS and only the object with highest confidence score is kept. \n\n\n**Q2–Discussing challenging cases and directions for future research:**\nThank you for your suggestions. Indeed, the BCI dataset is especially challenging because of the structure of the forest, including overlapping crowns; the lower resolution and quality of the imagery; and the long-tailed class distribution as highlighted in  5.1 and figure 4). We will provide further details and illustrative visualizations of predictions on the different datasets in the paper. Please see our response to Q3 for more on this topic, which is related to analysing errors of the models. \nThank you also for your suggestions for improving the discussion section. We will provide a more comprehensive discussion on directions for future research, expanding on the ideas presented in the conclusion (L359-372). In particular, regarding exploring other ways of using the DSM, for the R-CNN based models, one could  consider models using DSM prediction as an auxiliary task rather than as an input to the model. \nA different avenue is making use directly of the 3D point clouds that are used to obtain both the DSM and the RGB orthomosaics.  \n\nWe also believe that adding contextual metadata into our models in the form of spatial information, spectral information, and the type of forest could improve the performance of tree crown instance segmentation models.  In fact, while we have so far trained models separately on each dataset,  as more high-resolution drone imagery data becomes available, it could be very interesting from an application perspective to learn representations on combined datasets at different resolutions, and location metadata could prove helpful, as it has been shown to in species distribution modelling contexts. Spectral information, available through e.g. satellite open-source programs, could also be added as input into the models even if it is not available at a resolution as high as that of the drone imagery, as additional signal. \n\n\n**Q3- “...what is causing error in the model for the different datasets”** \nWe already provided a number of elements for analysis on the Quebec plantations dataset but will make sure that we include more details and  additional visualizations on other datasets as well. Regarding the Quebec plantations dataset, as illustrated in row 1 of Fig. 3, a case that is consistently missed across models is presented when trees crowns are very close to each other or don’t stand out much from the background. We have also observed errors in sites with tall herbaceous vegetation. As highlighted in the discussion, some classification errors align with human errors (e.g. distinguishing white and black spruces)  \nAcross datasets, overlapping crowns are indeed challenging. As an ablation, we trained a Mask R-CNN with DSM inputs only (no RGB image) to assess its informativeness on the Plantations dataset vs the SBL dataset. \n| **Dataset**   | **mAP single class** | **mIoU** | **mAP** \n| ---- | ----|----|----|\n| Plantations  | 0.464 |0.734 | 0.178\n| SBL | 0.092 |0.289 |0.023\n\nWhen looking at the drop in performance compared to the Mask R-CNN models using RGB images as inputs, we observe a much larger drop on the SBL dataset compared to the Plantations dataset. In fact, single-class mIoU remains high, which is another way to see that tree crowns are distinguishable from the background and from each other using the DSM only, and points to the usefulness of the DSM in plantation contexts. \n\n**Q4- application-readiness** – Our study shows that using the DSM along the RGB imagery consistently improves segmentation and classification results for the plantation use case, therefore, we would recommend that practitioners looking to quantify the carbon stored in boreal plantations include the DSM information in their models. We leave it to practitioners to decide, based on their application and available data which models to use. For example, if one only has access to bounding boxes as annotations, we showed that Faster R-CNN+SAM is a reasonable baseline to consider and that including the DSM helped (on both the SBL and Plantations datasets). \nWe will improve the discussion section with your feedback and outline more clearly future work directions with elements from our responses to Q2 and Q4 as well. \n\nThank you again for your time and reviewing our paper, and please let us know if you have any other questions."}, {"review_id": "z3fNxgGBJV", "reviewer": "Reviewer_QGDS", "summary": "The paper presents a tree crown segmentation approach based on SAM, adapted into a dedicated architecture. The model is evaluated across three datasets with diverse biomes. The authors incorporate photogrammetry-derived DSM as an additional modality and conduct extensive experiments and ablations. The study concludes that SAM out-of-the-box is ineffective for this task, prompt learning dramatically improves performance, and adding DSM information results in marginal but consistent improvements.", "questions": "**Questions:**\n- To run an RGB-only version of BALSAM, consider running experiments with VHR canopy height prediction models such as:\n  - [1] *Depth Any Canopy*, ECCVW 2024\n  - [2] *Very high resolution canopy height maps from RGB imagery*, RSE 2024\n  - [3] *Open-Canopy*, CVPR 2025\n- Can the photogrammetry pipeline to recover the DSM be elaborated further?\n- Weighted entropy loss shows no benefit; alternative losses like focal loss could be tested, possibly integrated into the hierarchical loss.\n\n**Remarks/Comments:**\n- The authors should clarify the terms of \"pretrained\" across tables and models to avoid misleading readers, as it refers to a specific part of a specific model, and can seem misleading as all models are pretrained to various degrees.\n- The figures would be so much better in vector format.\n- Tables are hard to read. Remove excessive digits, inconsistent spacing, and unnecessary parentheses around standard deviations.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n\n- Clear and well-written manuscript.\n- Extensive and well-designed experiments.\n- Thorough ablation\n- Evaluation on three datasets across diverse biomes.\n- Fair and appropriate baselines\n- Thematic and insightful analysis of results.\n\n**Weaknesses:**\n\n- The splitting strategy for the Quebec Plantations dataset may introduce spatial autocorrelation due to training and testing on the same sites; a site-wise split would be more robust.\n- The DSM modality only brings marginal improvements (~0.5%) and does not enable the model to consistently outperform simpler alternatives like RSPrompter.\n- The model comparisons are somewhat unfair, as not all baselines use the DSM; a canopy VHR height estimation model could have been used for RGB-only scenarios (e.g., [1,2,3]). At least the authors are very clear on which model has access to what\n\n**Overall opinion:**\n\nThe reviewer appreciates the solid effort and thorough evaluation, although the core contributions and findings are not particularly ground-breaking. The study is methodologically sound and offers useful empirical insights, especially regarding the limited benefit of DSM and the effectiveness of prompt learning. Overall, the reviewer likes the paper for reporting solid, honest, not exactly ground-breaking but ultimately useful results on a difficult and overlooked task.", "was_revised": true, "final_justification": "The paper is interesting and original, and my last concerns were addressed.", "rebuttal": "Dear reviewer, \n\n\nThank you for your thoughtful review and helpful comments. \nWe are pleased to hear that you appreciate the quality of this work and the importance of the problem we are tackling  and that you highlighted that you _“[like] the paper for reporting solid, honest, not exactly ground-breaking but ultimately useful results on a difficult and overlooked task.”_\n\n\nWe respond to your comments and questions below. \n\n\n**W1- Splits:** We will make the explanation on splits clearer in the paper. First, we would like to clarify that there are neither image pixels, nor annotations that belong to two different splits. Furthermore, while some sites are represented in both the testing and training sets in the Quebec Plantations dataset, we tried as much as possible to assign sites fully to splits and limit spatial autocorrelation. The only reason why some orthomosaics were divided into both the training and test sets is the distribution of species across sites. Indeed, we wanted to make sure that species of interest were both in the train and test set. This is the only time when we manually drew AOIs. However, as much as possible, the train/val/test regions were kept as spatially separate “blocks” ensuring no overlap between the splits. The table below shows in which splits the sites are assigned. Only the sites afcagauthmelpin and afcagauthier were broken up into the train and test sets. We will add a visualization in the Appendix of the splits for both the training and testing sets.  \n\n\n|Site  | **Train** | **Val** | **Test** |\n|---|---|---|---|\n| cbpapinas |  | x | x |\n| cbblackburn1 |  | x |  |\n| cbblackburn2 | x |  |  |\n| cbblackburn3 |  |  | x |\n| cbblackburn4 |  |  | x |\n| cbblackburn5 | x |  |  |\n| cbblackburn6 | x |  |  |\n| cbbernard1 | x |  |  |\n| cbbernard2 |  | x |  |\n| cbbernard3 |  | x | x |\n| cbbernard4 | x |  |  |\n| afcamoisan | x |  |  |\n| afcahoule | x |  |  |\n| afcagauthmelpin | x | x | x |\n| afcagauthier | x | x | x |\n\n\n**W3/Q1-Q2 - re an RGB-only version of BalSAM  and explaining the photogrammetry pipeline to recover the DSM** : \nWe understand your suggestion of running an RGB-only version of BalSAM as a suggestion to use canopy height estimation from RGB images instead of the DSM as input to the model. We would like to clarify that the DSM is already a product derived from RGB imagery only, and is always available at no additional cost if drone RGB imagery is available. As a consequence, it is also much higher resolution than the imagery used by the models in the papers linked in your review. For example, the DSM on the Plantations dataset is 0.5cm/pixel resolution, while HRV canopy height maps are typically ~0.3m. \n\nStructure-from-motion (Sfm) photogrammetry, which is used to create RGB orthomosaics from high-resolution RGB drone imagery, generates 3D photogrammetry dense point clouds, from which the DSM is derived. \nIn essence, the approach is to:\n1. Find key points in individual images (SIFT).\n2. Match keypoints in image pairs to estimate camera locations and parameters (SfM).\n3. Densify the 3D point cloud (MVS).\n4. Project that point cloud to 2D to get the DSM.\n\nWe refer the reviewer to [1] for further details on the photogrammetry pipeline.\n\nFig 1. of this paper shows that the DSM is just a 2D projection of the dense point cloud, with interpolation to fill missing values when projecting the 3D points to a 2D grid (raster).\nTherefore, in some sense, BalSAM is already based only on RGB imagery. Please let us know if that answers your question.\n \nAs for the comment that “model comparisons are somewhat unfair, as not all baselines use the DSM”. Could you kindly clarify your concern? One of the axes of this study is to determine when and how the DSM is useful, which is why we explicitly compared the same methods with and without DSM input, as shown in our results tables 1, 2 and 3 with the checkmarks in the column “DSM”.\n\n\n**W2. “The DSM modality only brings marginal improvements and does not enable the model to consistently outperform simpler alternatives like RSPrompter.”** A main goal of this study was actually to investigate whether the DSM helps for certain forest contexts and architectures. Our study shows 1/ that using the DSM alongside RGB imagery improves segmentation and classification mean results in the case of tree plantations, which have a specific structure, 2/ in mature forests, the DSM either improves results or keeps them comparable within error bars, 3/ while SAM out-of-the-box does not perform well, SAM-prompt based methods (RSPrompter, BalSAM), which have so far not been used for tree crown segmentation, reach best performance overall on all the datasets. \nThis work is a first step towards guiding practitioners in their choices of models for forest monitoring, where different use cases may call for different methods depending on the nature of the data. \nBalSAM only has an additional 10M parameters compared to RSPrompter, which is a fraction of the total model size (the SAM image encoder alone has 632M parameters). \n\n\n**Q3 – re the focal loss:** \nUpon your suggestion, we ran an **experiment with Focal loss with Mask R-CNN on the SBL dataset** (to be consistent with the setup of results reported in table 14 Appendix D.4.) The gamma parameter of the loss was set to 2. Results are reported below: \n| mAP single-class | mIoU single-class | mAP | wmAP|\n|---|---|---|---|\n|30.6|63.9|14.8|23.7|\n\n\nThe model performs better than with the Weighted Loss, but not as well as when using Cross-entropy loss. We are currently running more experiments with other values of the gamma parameter. \n\n\n**Remarks: “[...] clarify the terms of \"pretrained\" across tables and models [...]”:** \nFor R-CNN based models, we explain in the text and tables captions that “pre-trained“ models refer to backbones initialized with ImageNet weights as opposed to trained from scratch. For SAM-based models (SAM, RSPrompter, BalSAM), we indeed use the weights from SAM but the image encoder and mask decoder are kept frozen in all configurations. We’ll make sure to highlight this better in the paper. \n\n\n**Formatting:** Thank you for the very helpful feedback, we took this into account and will change figures to vector format and format the tables in a clearer way. \n\nMany thanks for taking the time to review our paper and please let us know if you have further questions. \n\n\n[1] _Iglhaut J, Cabo C, Puliti S, Piermattei L, O’Connor J, Rosette J (2019) Structure from Motion Photogrammetry in Forestry: a Review. Current Forestry Reports 5: 155–168._"}]

Author rebuttal:

Meta-review: This paper presents a tree crown segmentation approach based on SAM using high-resolution imagery across different geographic contexts. Specifically, the paper proposes BalSAM, which utilizes the RSPrompter framework to incorporate Digital Surface Model (elevation information) data for improved segmentation. The model is evaluated across three datasets with diverse biomes and compared to a range of other segmentation approaches, including SAM, prompted SAM, Mask R-CNN, and RSPrompter—across three distinct forest environments: boreal plantations, temperate forests, and tropical forests (datasets with diverse biomes). The final results show that current baselines perform poorly and that the integration of DSM data shows marginal but consistent improvements, particularly in the structured context of plantations.


Strenghts

S1. The paper is well-written and easy to follow.

S2. The introduction motivates tree crown delineation problems, the significant problems in forest monitoring, and the challenge of segmenting and classifying individual trees from UAV imagery.

S3 The integration of SAM with DSM height to improve tree segmentation. Moreover, both use only RGB drone imagery, so they do not require other sensors. 

S4. The experiments were extensive, considering three different geographic datasets and multiple baselines. 

S5. The experiments show a slight but consistent improvement with respect to the baselines.


Weaknesses

W1. There are deep concerns about the significance of the results. They are slightly better, but not always consistent. 

W2. The inclusion of DSM into SAM is not a very novel work.

W3. The evaluation could be unfair, considering that not all the baselines use DSM.

W4. The baselines could be too complex, and other models could be considered, such as Mask2Former, the latest Transformer-based instance segmentation, or fine-tuned SAM.

W5. The standard errors of the proposed model are relatively large, which may indicate that the results are not statistically significant.

W6. The splitting strategy on Quebec datasets could introduce spatial autocorrelation.

W7. The discussion and ablation sections were quite general. It would have been useful to offer suggestions about the generalization of BalSAM.

Despite significant weaknesses in novelty, experiment fairness, and result significance, most reviewers push for acceptance, with only one reviewer offering a weak rejection. Three reviewers considered that the problem is interesting, and the framework is novel.

The discussion included a question regarding the trade-offs between accuracy, inference speed, and training cost, which was addressed. However, one reviewer was still not convinced that the inclusion of the DSM generates an important contribution to the model. In general, the authors agree with some of the weaknesses, such as significant standard error and slight improvement in the error.
Decision: Accept (poster)