Mastering SAM Prompts: A Large-Scale Empirical Study in Segmentation Refinement for Scientific Imaging
Authors: Stephen Price
Elke Rundensteiner
Danielle L. Cote
Venue: Accepted by TMLR ()

Abstract: Segment Anything Model (SAM) has emerged as a prevalent tool empowering advances in vision tasks from instance segmentation, panoptic segmentation, to interactive segmentation. Leveraging powerful zero-shot capabilities enabled by visual prompts such as masks placed on the image, SAM has been shown to significantly improve tasks. Yet, a poor prompt can worsen SAM performance, risking consequences such as misdiagnoses, autonomous driving failures, or manufacturing defects. However, recent studies on visual SAM prompting remain limited, cover only a small fraction of potential prompt configurations, adopt ad-hoc evaluation strategies, and come with limited or even no rigorous analysis of the statistical significance of prompt configurations. To address this gap, we undertake the first large-scale empirical study comprehensively evaluating the impact of SAM prompt configurations on segmentation refinement. This includes 2,688 prompt configurations, including points, boxes, and masks with diverse augmentations, on four initial segmentation models for a total of 10,752 evaluations. From these results, we draw statistically significant insights along with practical guidelines for prompt design on scientific images. In particular, we recommend including a bounding box, which raised AP@50-95 by 0.320 and advise against using a coarse mask, which lowers AP@50-95 by -0.133 across all four models on microscopy data sets.  We showcase that our recommended prompt configuration enables SAM to outperform leading refinement methods on multiple scientific benchmark datasets.

Published in Transactions on Machine Learning Research (11/2025)
Mastering SAM Prompts: A Large-Scale Empirical Study in Segmentation Refinement for Scientific Imaging
Stephen Price
sprice edu
Department of Computer Science, Worcester Polytechnic Institute
Danielle L. Cote
dleote Seopi edu
Department of Mechanical and Materials Engineering, Worcester Pokytechmic Institute
Elke A. Rundensteiner
rundenst@upii edu
Department of Computer Science, Data Science AT Program, Worcester Polytechnic Institute
Reviewed on OpenReview: https: // openrvieew.. net/ farm///rmmm id= www.:e/eeeuuppptttttttoeeeSaaea
Abstract
Segment Anything Model (SAM) has emerged as a prevalent tool empowering advances in vi- sion tasks from instance segmentation, panoptic segmentation, to interactive segmentation. Leveraging powerful zero-shot capabilities enabled by visual prompts such as masks placed on the image, SAM has boen shown to significantly improve tasks. Yct, a poor prompt can worsen SAM performance, risking consequences such misdiagnoses, autonomous driving failures, manufacturing defects. However, recent studies visual SAM prompting main limited, cover only small fraction of potential prompt configurations, adopt ad-hoc evaluation strategics, and come with limited or even no rigorous analysis of the statistical significance of prompt. configurations. To address this gap, we undertake the first. large- scale empirical study comprehensively evaluating the impact of SAM prompt configurations on segmentation refinement. This includes 2,688 prompt configurations, including points, boxes, and masks with diverse augmentations, on four initial segmentation models for to- tal of 10,752 evaluations. From these results, we draw statistically significant insights along with practical guidelines for prompt design on scientific images. In particular, recom- mend including a bounding bax, which raised AP@50-95 by 0.320 and advise against using a coarse mask, which lowers A AP@50-95 by -0.133 across all four models on microscopy data sets. We showcase that our recommended prompt configuration enables SAM to outperform leading refinement methods on multiple scientific benchmark datasets.
1 Introduction
The Promise of Segment Anything Model (SAM). The recent introduction of Segment Anything Model (SAM) by Kirillov et al. (2023) has revolutionized the field and practice of computer vision, streamlin- ing numerous tasks that had previously been significantly more challenging. SAM has enabled improvements in weakly supervised instance segmentation (Wei al. 2024), high-resolution object segmentation (Ke al., 2024), zero-shot segmenting (Yamagiwa al.) 2024), and 3D-object detection (Zhang et al. 2023b). Impres- sively, SAM leverages zero-shot prompting to achieve these advancements across diverse range of tasks. User-specified visual prompts, such as points, boxes, coarse masks, guide the model's outputs to the de- sired task without requiring additional training or prohibitively expensive modification to the massive model architecture. In this work, we leverage SAM for segmentation refinement. of scientific and medical microscopy (Figure Iap, converting initial segmentation masks into prompts and producing high- quglitualty refinements. We focus on refinement because supports rapid crcation of large numbers of input masks, enables automated prompt derivation per instance according to defined criteria, and allows high-fidelity quantitative evaluation against ground-truth IoU.

---

Published in Transactions on Machine Learning Research (11/2025)
(a)
(b)
Figure (a) Example segmentation refinement pipeline using SAM, including making initial predictions, processing them into prompts for SAM, producing refined segmentations, and aggregating across all objects. (b) Impact of example prompl confiuraaions SAM segmentations, showcasing how variations drasti- cally alter final segmentations (red dots denote a prompt placement of points visual prompts).
Limitations of SAM. Despite extensive work integrating (Wei et al. 2024), extending (Ke et al. 2024 or fine-tuning SAM (Wu et al. 2025), there is lack of systematic evaluation of how prompt design affects its performance. While SAM has the potential for strong zero-shot capabilities, SAM's accuracy is highly dependent on the "quality' of its prompt. For example, a single point-prompt might cause SAM to under- segment the desired object, as shown Figure [b] By contrast, using multiple point-prompts can improve coverage of the object's boundaries but requires careful selection of point locations but again too many or poorly placed points can inadvertently force SAM to include surrounding regions merge adjacent. objects (Wei et al. 2024 Additionally, SAM's performance is demainndependennt, often underperforming contrast, fine- graiigd scientific and medical microscopy images essential for critical applications (Ma et al. 2024h. In this work, we therefore concentrate on prompt design for this important class of scientific and medical applications that leverage microscopy images to derive practical guidelines.
low-
Research Questions. Given the variability in image resolution, object morphology, and desired prompt attributes, the number of unique prompts is near-in wear-infinite. As result, open questions remain regarding prompt design composition. In this work, conduct rigorous study of performance trends to guide prompt selection and answer the following rescarch questions in microscopy settings: What is the relative importance of boxes, masks, and point prompts segmentation quality? How do different potential lypes of visual augmentation strategies (e.g., point-placement strategies) impact SAM's results? How does combining different prompt types and their visual augmentations affect segmentation quality, in particular, what are the challenges and promises derived by the polential interaction (amplification, attenuation, or even cancellation out of the positive or negative impact) of one prompt type on others when combined into visual prompt configurations? Finally, with SAM's training set, SA-1B, primarily including natural-scene high-contrast benchmarks such COCO Cityscapes rather than the low-contrast, small-scale, and irregular features common in scientific and medical images (Li et al. 2025 if and how best. can SAM's performance be effectively adapted to the unique challenges in these scientific and medical domains?
State-of-the-Ar and its Shortcomings. Consequently, recent studies have found that SAM's out- of-the-box performance often falls short. on segmentation tasks in these important scientific and medical application domains (Ma al., 2024] Zhang et al.} (024c).. With the growing demand for generalizable yet. high-precision segnentation models in these important domains including cancer detection (Kassis et al. 2024), tigh-throughhut microscopy (Rusanovsky et al., 2022), histopathology (Sikaroudi et al., 2023 and surgical planning MMin et al. 2025), establishing effective visual prompting strategies to achieve high- quality segmentation for these settings is imperative. Without an effective prompting strategy, these works implemented alternative strategies, such as retraining SAM (Ma et al. 2024), relying on human-in-the-loop
interactive segmentations (Shen et al. 2024), or both CCheng et al. 2023b).
Recently, Cheng al. (0033 Mayladan al. (002)), and Dai et al. (2023) have begun to explore SAM prompting across natural, medical, and remote-sensing data. However, these studies have been exceedingly
2

---

Published in Transactions on Machine Learning Research (11/2025)
small, typically evaluating fewer than 25 prompt configurations. With few combinations, these studies have not statistically validated the significance of each prompt type or of its potential types of visual aug- mentations. Lastly, these works typically explored a single prompt type (points, bounding boxes, and coarse masks) at a time, overlooking the problem of determining the complex interactions of prompt types and their augmentations when combined into visual prompt configurations.
Our Approach: Large-Scale Analysis of Prompt Design Configuration. To address this gap, we undertake comprehensive study of visual prompt design configurations and their combined impact the performance of SAM for segmentation refinement in scientific images. Our rigorous study covers 2,688 prompt designs on four alternate popular initial segmentation models, resulting in 10,752 evaluations. In particular, we first identify the effect of each core prompt type (points, masks, and boxes) and their unique augmentation strategies. Namely, for boxes, uniform scaling is employed to expand or contract the as-detected boundary while preserving spatial location. For masks, a similar approach is considered, but with additional requirements to ensure the preservation of the object's morphology. For points, we evaluate multiple strategies, including adjusting the number of points placed, tailoring the search space for placement, and modifying the placement algorithm. Next, we also thoroughly explore their prompt type interrelationships by creating composite prompt configurations that include (or exclude) cach core prompt type, as well as their respective augmentations.
Throughout this evaluation, we discovered significant prevalent trends that serve as a foundation for deriving guidance for practitioners for prompt design. For example, prompts that contained bounding boxes improved performance by 0.320 AP@50-95 through a wide array of testing scenarios on scientific images. Alternatively, adding coarse masks reduced it by -0.133 AP@50-95, and point-based components could boost accuracy but were dependent generality of these trends, we then extended our analysis of the top prompt's performance to determine its impact on other metrics, including individual IoU thresholds (AP@50, AP@75, AP@95), revealing both bulk and fine-grained improvements. We further explored the robustness and generalizability of our cvaluation by applying our lop-performing prompt configuration developed on one key data set as is, without further prompt refinement, to other benchmark scientific datasets not considered during the prompt design study, For this, we then compare SAM's zero-shot refinement performance using our identified recommended prompt configuration against the leading model-agnostic segmentation refiners, CascadePSP (Cheng et al. 2020) and SegRefiner (Wang et al., 2024h. Overall, our study highlights that SAM, with our well-designed prompt, can achieve significant improvements in segmentation quality on scientific images, domain where it has previously boen found to struggle (Ma et al. 2024).
the number of points and their spatial placement. To validate the robustness and
Contributions. In summary, this work offers the following contributions:
The largest empirical study of SAM prompt design date, covering 2,688 unique configurations and four initial popular segmentation models, totaling 10,752 evaluations. This study covers the first sys- tematic analysis of how points, boxes, and masks interact when combined in prompt configuration. Comprehensive evaluations demonstrating statistically significant trends for prompt design, leading to valuable guidelines for practitioners for the utilization of SAM in scientific and medical domains. An experimental validation of SAM's model-agnostic performance compared to state-of-the-art tech- niques, particularly designed to tackle the segmentation refinement problem across three scientific image datasets, four initial segmentation models, and four performance metrics.
2 Related Works
2.1 Segmentation Refinement Approaches
Segmentation refinement refers to the task of processing an initial model's output to yield an improved new mask (Tang et al. 2021). These methods fall into two categories: model-dependenn and model-agnostic solutions. Model-dependent solutions, such as RefineMask (Zhang et al. 2021). PointRend (Kiilllvv et al., 2020), and Mask Scoring R-CNN (Huang et al., 2019), add additional layers a model's architecture produce higher resolution or more accurate segmentations. However, these approaches are tailored a specific model architecture and, as a result, typically cannot be used elsewhere. Model-agnostic approaches,
3

---

Published in Transactions on Machine Learning Research (11/2025)
such CascadePSP (Cheng et al. 2020 SegRefiner (Wang et al. 2024), instead post-process an initial model's segmentation. The latter eliminates the dependency on the specific architecture, offering improved versatility across domains. For this reason, we henceforth focus on the latter approach.
Model-Agnostic Segmentation Refinement. Prior to the introduction of SAM, model-agnostic segmen- fation refinement approaches were not based on prompting. One of the first refinement methods, CascadePSP (Cheng et al., 2020). applies cascade of boundary-aware residual modules on a coarse mask, progressively enhancing local edge detail. SegRefiner (Wang et al., 2024) at NeurIPS 2023 was the first to integrate diffusion-based models into the refinement process, building on previous work using GANs (Le 2020 Most rocently, Meta's Segment Anything Model (SAM) (Kirillvv et al. 2023). a foundational computer vision
model, has emerged as a highly competitive segmentation refiner (Lin et al. 2025 Yu et al.) 2024 Wei et al.
2024 Chen et al.) 2024 Mayladan et al. 2023). When using SAM, the initial segmentations are used to derive visual objects that serve as prompts, such as bounding boxes, coarse masks, and points, to aid SAM's ro-sogmentation (See Figure la].
2.2 SAM and Visual Prompting
Previous works, while starting to analyze the effect of prompting on SAM performance, have been quite limited. In particular, prompt types are frequently analyzed in isolation. For example, Hu et al. (2023) briefly tested how increasing the number of points from 1. 5, 10, or 20 affected performance but did not specify how points were placed and ignored any other prompt types. Alternatively, Cheng et al. performed a preliminary analysis of six bounding box prompts and three-point prompts. However, there was no variation in the point-prompting method. They neglected to evaluate when points and boxes are used together. Dai al. (2023) explored four point-placement strategies but constrained their experiments to only two points. While they explored few box prompts, similarly to prior work, they did not explore points and boxes integrated together into one unified prompt configuration. Lastly, Mayladan et al. (2023) explored two prompts containing a box and points testing one and five points only, plus, no placement strategy was indicated.
2023a
Our Work in Comparison. These experiments, demonstrating the need for further analysis on prompting,
were ad hoc in nature. They contained too few samples (often less than ten) to evaluate statistical significance
CCeng et al.
2023a Hu et al.) 2023 Mayladan et al.] 2023). rarely explored multiple prompt types together (0233a):
Cheng et al.
; Dai et al. (2023); Hu et al. 2023), and frequently did not give enough information to
reproduce results or analyze trends, particularly with respect to how points were placed (Cheng et al. 2023a
Hu et al. 2023 Mayladan et al. 2023). In comparison, our work addresses each of these shortcomings. We offer the first large-scale evaluation of prompts, containing 2,688 distinct prompts, enabling the statistical significance of each prompt type and each prompt augmentation method to be evaluated. We performed the first systematic cross-promin pross-prompiig evaluation, identifying each prompt type's effect on the others. Additionally, we evaluate the effect of coarse masks, which were not explored in these previous works. Lastly, we introduce iterative refinement, testing how SAM's refinement capabilities change over multiple iterations.
2.3 Existing Surveys on Target Domains for SAM
Prior works, such as Ji et al. 2024), discussed SAM's performance and potential application areas. However, these works primarily reviewed where SAM works, and focused less on how to improve SAM. For example, surveys such as ZZhang et al. (2023a) and Zhang et al. (2024a) extensively have cataloged SAM's usage and performance across diverse domains, including image- image-based (asks, human robot interaction, and more specialized fields such as remote sensing. While these studies offer a useful overview of potential domains for SAM-based application and rescarch, they provide little guidance on improving performance or insights into the stability of results. In contrast, our work specifically targets methods for improving performance, particularly on scientific images, domain where SAM is known to struggle (Ma et al. 2024).
4

---

Published in Transactions on Machine Learning Research (11/2025)
3
Constructing SAM Prompts
SAM was trained on the SA-1B dataset, containing 11 million images with over 1 billion masks, making it a highly capable and one of the most popular segmenters (Kirillov et al. 2023 While SAM is skilled at detecting contours, it demonstrates semantic ambiguity, occasionally assigning higher confidence undesired sub-regions (e.g., t-shirt instead of person, or sub-component of particle as shown in Figure D To overcome this ambiguity, well-designed prompt must accurately delineate the intended object, providing full coverage to prevent erroneous annotations of sub-regions, yet not expansive that it merges the object with its neighbors surrounding error.
When constructing a prompt, SAM takes two types of inputs: sparse (points, boxes, and text) and dense (mask), as categorized Figure 2 locations or regions for SAM to process, while text. prompts are non-spatial by offering semantic guidance. In our analysis, we cover all types of spatial prompts. addition, we tested numerous prompt engincering strategies in the form of visual prompl augmentations in an effort to represent the desired object more closely. For example, evaluated the impact of shrinking or enlarging boxes and masks. For points of interest (POIs), the augmentation steps are more complex due to their interconnected nature. Specifically, among the three augmentation factors for points (number of points, placement algorithm, and perimeter buffer), we note that the actual placement depends jointly on all three, since the buffer modifies the available search space, and the number of points augments the decision process each algorithm.
Of these, boxes, points, and masks are spatial prompts, specifying exact
Boundary
Shrink
Box
Sparse
Enlarge
Points
POI Placement
Algorithm Selection
Prompt
Number POIs
Perimeter Buffer
Text"
Coarse
Dense
Shrink
Mask
Enlarge
Figure 2: Characterization of prompt types and augmentations for SAM-based segmentation refinement.
3.1 Bounding Box Visual Prompt and Its Augmentations
Using predictions from an initial model, which we aim to refine, a bounding box B can be extracted for each detected instance. This box is the tightest boundary area containing the instance mask, represented by the v, y coordinates of the upper-left corner, the width, and the height (Eq. D. This box can be used as a prompt for SAM as-is without any augmentation. However, it can also be augmented by scaling factor, a, produce B', described in Equation 3 This is achieved by uniformly scaling the width and height by Va, creating w' and W', and adjusting and by half this change, producing z' and y', described in Equation 2 The resultant augmentation produces bounding box whose area is scaled by some factor while the center is held constant, preserving the spatial location of the segmentation. This augmentation can be useful in cases of poor initial segmentations, enabling expansion for under- inder-segmentation cases or contraction for over-segmentation cases.
B (x, y, w. h)
(1)
w/ = vo=vaw w.
= =a h
d' = r +
Va
w
D' == +
(1 Voj
(2)
2
2
B' a. B (x', y' w h²)
(3)
5

---

When evaluating POT augmentation strategies like POT placement, we aim to balance two competing inter- ests: spacing the POls far enough apart to fully cover the desired object, while keeping them close enough to avoid false prompts outside the object. To address this, we distinguish between and then evaluate three prompt placement parameters: the number of POls (denoted as k), the perimeter exclusion area (denoted as Published in Transactions on Machine Learning Research (11/2025)
3.2 Coarse Mask Visual Prompt and Its Augmentations
Like the bounding box, coarse mask M C Ω (where Q C Z is the set of all image-pixel coordinates) can be given to SAM as detected, or can be augmented by some factor À to produce M', an enlarged or shrunk version. However, as a complex shape, M cannot be uniformly scaled in the same way while preserving the spatial location and morphological characteristics the object. Instead, M must be augmented at the pixel level, applying erosion (0) to shrink and dilation (#) to enlarge it by X. First, we define a discrete disk r(A) such that eroding or dilating M by De yields mask whose area is approximately AMM,, as shown in Equation 5 Finally, we define M' M, with the inclusion of any addit ional pixel contained within the disk D, centered at some pixel in M for dilation or M after excluding any pixel contained within the disk D, centered at some pixel outside of M for erosion, as shown in Equation G Through this transformation, we preserve the spatial location and a similar morphology.
with radius i, as shown in Equation 4 Next, define the smallest
EZ; Dj = (pe.Pa) E Z² p2 + p2 S t]
(4)
min IEN: M D. 2 AlM] 121,
r(A)
(5)
min min tEN: M Di S A 0<d<1.
M 2 DA(A) = : M, de Dr()) with u= u+++d A21,
M'
(6)
M e D((A) = {uER: 3 E M, d Dr() with u == u-=-dd d]. 0<A<1.
3.3 POI Placement Augmentations
7), and the POI placement algorithm (denoted as Φ). In this work, explore values of k ranging from 1 to 7. We evaluate y as mitigaling factor to prevent POIs from being placed too close to the object boundary, where false prompts are more likely. In our experiments, we apply a perimeter buffer 7 of 0%. 5%, 10%, and 15%, eroding the original mask M to yield the search region M' with Equations 4 Note the distinction between M' and M': the paramcter y only defines the eroded region used for POI placement, whereas X governs how we dilate or erode the mask input itself.
With the number of POls k and the POL search space defined by 7, we design and then evaluate three POI placement algorithms (Ø): random, distance maximization, and Voronoi placement. The three algorithms are described below.
Random Placement. For random placement (Alg. D number of points & are taken as inputs. Once M croded by y to produce M defining the search space, we uniformly sample k points.
the initial mask M, perimeter buffer 7, and
Distance Maximization. We design a second algorithm, called distance maximization, with the aim to address two issues that can arise from random placement, namely: (i) points landing too close to each other, and thus likely to add little additional value, and (ii) sub-components of object left unprompted due to all points being concentrated elsewhere. The intuition is to maximize pairwise distances between points, we avoid point crowding, broaden spatial coverage, and reduce the likelihood that any object subregion is missed. To achieve this (Alg, 2), the algorithm accepts the same inputs. Further, we define M the same as for random placement, and V as all points contained within M For cach C C V such that |C] k we evaluate the total pairwise distance between all points p C, and return P as the set C with the largest total pairwise distance.
6

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm 1: Random Point Placement
Input: Binary mask M, perimeter buffer 7, number points lc Output: Selected points P
M & M e YY);
V - y(r,)) M*);
P+00: 0;
while |P] < k do
pick (z. y) uniformly at random from V;
p P {(r,1));
V {(,,8);
return P:
Algorithm 2: DistanceeMaxxiizzaiio Placement
Input: Binary mask M, perimeter buffer y, number points
Output: Selected points
M M rr
V + (z) pixel (2) is on the boundary of M*) };
maxDistance -00;
P <- 0:
for each subset C C V with |C| = k do tempDistance + totalPairwise DistarrreseeDsttneeee( (C))
if tempDistance > maxDistance then maxDistance +- tempDistance;
C
return P;
Voronoi Placement. Our third algorithm, Voronoi placement, continues to avoid point crowding and to maximize coverage. However, in addition, the algorithm aims to reduce prompting outside of the true object- an issue that can arisc with distance maximization. The Alg. 3 again accepts the same inputs and defines M as above. To achieve the above goal, however, it now approximates a centroidal Voronoi Lessellation of M into cells via Lloyd's k-means iterations. The algorithm then returns the centroid of each cell as the POL placement.
Algorithm 3: Voronoi Point Placoment
Input: Binary mask M, perimeter buffer ?, number of points k; iterations I Output: Selected points P
M + -M V ((r,y) E M*};
Initialize ci...., Ck by sampling k distinct points from V; for i i to do
foreach E V do assign to cluster j arg min;|/u -iai:://lu///00 call2:
for i + to do
L 4 f mean (fu V v assigned to i));
P +- {r)] return P:
With random placement as baseline, Voronoi and distance maximization use the same underlying strategy for placement, assuming that points farther apart will be more representative of the complete mask. However,

---

Published in Transactions on Machine Learning Research (11/2025)
since Voronoi places points at the centroid of each cluster, these points not as likely to be placed on the boundary and will also be spread around the inner regions of the mask more, resulting in a distinct approach. Piecing this together, the set of POIs is determined by the imitial mask (M), the number of POIs (k), the chosen perimeter buffer (7), and the selected algorithm for placement (Φ), as Equation
Ps f(M, o, k;
3.4 Final Prompt Composition
Ultimately, the final prompt, P, corresponds to the combination of the desired components after augmenta- tion, as described in Equation and visualized in Figure 3 Since prompt can contain any combination of visual prompt types, represent the inclusion (or exclusion) of each component with oB, êM, and öp binary flags. Here, dB indicates if the box B' is included, dы specifies whether the mask M' is included, and 6p determines if the generated POIs, Ps, are incorporated. As shown in Figure 3 after all necessary aug- mentations have been performed, the desired inputs are joined together and fed to SAM for re-segmentation, producing a final refined segmentation.
B' dA M', 8p Ps SB.0M,8p E {0,
(8)
Bounding Inclusion (5g)
Initial Box
Final (B7
Distortion
Segmant-Arything Re-Sagmenting
Prediction
Coarse Mask Inclusion (5.,)
Initial Mask (M)
Finell Mask (M'
Distortion (
POI Inclusion (8p)
Initial Maak
Algorithm (+)
Final POls
(y)
(K)
Required Pracess
Optional Prompt
Initial Model Selection/rrciccton
Box Prompt
Mask Prompt
POl Prompt
SAM Refinement
Figure 3: Framework of prompt input selection and augmentation considered for SAM-based refinement.
4
Experimental Methodology
4.1 Datasets
For the primary analysis, we used microscopy dataset composed of micrographs from a scanning electron microscope of powder particles used in additive manufacturing. In powder-based additive manufacturing, particle size and shape are critical to overall build quality. Particles too large can leave porous regions, negatively impacting the st.uuctural integrity of manufactured parts. Similarly, particles with irregular mor- phology can affect the flowability of a powder, decreasing the deposition efficiency and increasing costs. As a rosult, there has been growing interest in high-throughhut powder analysis frameworks (Cohn et al. 2021
Price et al. 2021 ). For the primary evaluation, used powder dataset containing 132 images and 7,324 annotations Price et al. 2025). To ensure generalizability, after identifying the optimal prompt configuration on the primary
partece dataset, we compared SAM's refinement performance using our identified prompt against stato-offthheart methods on two additional publicly available microscopy datasets: a general collection of 385 images covering particles, grains, and cells with 8,877 annotations (Tonneau, 2023), and a cellular microscopy dataset of 65 images with 2,193 annotations Yuki 2024).
4.2 Models
Vision Models for Initial Segmentation. Four vision models were used for initial segmentation: YOLOv8 Nano, YOLOv8 X-Large [Cltralytics 2023), Mask R-CNN (He et al. (2017). and Mask2Forrer Cheng al. 2022 These models were chosen for their growing popularity in applied scientific domains

---

Published in Transactions on Machine Learning Research (11/2025)
Zhu al. 2023
Zhang et al. 2024b d Tang et al. 2025 Wankhade al. 2024 Cohn et al 2021
Price et al. 2021
These models also cover diverse range of model sizes and architectures, including
convolutional laaeriooon---edddddd and transformer- backbones.
Vision Models for Segmentation Refinement. For SAM refinement, we used the ViT-L backbone (Alexey 2020) with the published model weights and parameters, specified in Kirillov et al. (2023) For refinement techniques, we employed CascadePSP, introduced at CVPR Cheng et al. 2020), in its default configuration. Similarly, we used SegRefiner, presented at NeurIPS
comparison against state-of-the
2020
2023
Wang et al. 2024), evaluating both its small and large variants.
4.3 Metrics
To evaluate performance, Average Precision (AP) was computed Intersection-over -ver--nno--n Union (IoU) thresholds of 0.50,, 0.75, and 0.95, and the average of all thresholds between 0.50 and 0.95 (in 0.05 increments). AP@50 provides the most lemient metric, where segmentation is considered correct if loU > 50%. In contrast, AP@95 provides the strictest metric, requiring the IoU 2 95% for a segmentation to be considered correct. AP@75 offers middle ground between these metries, while AP@50-95 uniformly summarizes performance across multiple thresholds. Mcasuring performance at various IoU thresholds enables us to track improvement at different stages of initial prediction accuracy, offering insight. into where refinement is most useful.
4.4 Experimental Design
In this work, our experiments can be broken into three main categories: prompt optimization, state-of-the- art comparison, and statistical significance testing. For prompt optimization, systematically evaluated 2,688 unique prompt configurations, encompassing combinations of points (number, placement algorithm, perimeter buffer), boxes (inclusion and scale), coarse masks (inclusion and prosion/dilations and iterative refinement, on four initial segmentation models, as outlined in Table [ For the state-of-the-art evaluation assessing generalizability of our best prompt derived on our first. dataset to CascadePSP (Cheng et al. 2020), SegRefiner Small and SegRefiner Large (Wang al.) 20024) on all four initial segmentation models and three microscopy datasets (powder, general microscopy, and cells). Lastly, to validate mcasured improvements, t-tests on individual prompt parameters (e.g., box no-box, various POI strategies, etc.) as well overall performance differences between SAM and alternate refinement methods.
our proposed prompt configuration design, we compared segmentation results
conducted paired
Table
Experimental conditions tested for prompt creation.
Category
EQ
Conditions
Model
YOLOv8 Nano, YOLOv8 X-Large, Mask R-CNN, Mask2Former
Box Inclusion
6
Yes / No
Box Distortion
a
90%, 100%, 110% (when included)
Mask Inclusion
8M
Yes / No
Mask Distortion
A
90%, 100%, 110% (when included)
POI Algorithm
Random, Distance Max, Voronoi
POI Perimeter Buffer
0%, 5%, 10%, 15%
Number of POIs
k
1, 2, 3, 4, 5, 6,
Iterative Refinement
One Stage / Two Stage
Rationale for Paramcter Ranges. When designing these experiments, our objective was to improve boundary quality while preserving the location and size of an object, established by the domain-trained base model. Accordingly, we limited box and mask adjustments to +10%. For the perimeter buffer, we sought reduce the likelihood of false points near uncertain edges without overpowering the placement algorithm. We thus varied perimeter buffer by y E (0%, 5%, 10%, 15%). Lastly, we also explored the effect of increasing the number of points. Howevvr, prior work in interactive prompting with SAM has shown that the optimal

---

Published in Transactions on Machine Learning Research (11/2025)
number of points was approximately seven or fewer before performance started to deteriorate with additional points (Quesada et al. 2024). Thus, in this work, we constrained & to maximum of seven points.
5 Experimental Results
In our prompt optimization experiments, identified multiple trends leading a well-designed prompt, which we discuss further in Section 5.1 From these results, determined the optimal prompt included bounding box (as-detected), no coarse mask, 3 POls, Voronoi placement, and 10% perimeter buffer, yielding crisper and more precise segmentation across all four initial segmentation models, as shown in Figure 4a Using this prompt, on the powder datasel, we found stalistically significant improvement in the AP@50-95 by 0.203, 0.201, 0.128, and 0.081 with << 0.0001 for YOLOv8 Nano, YOLOv8 X-Large, Mask R-CNN, and Fask2Former, as shown Table 2 Compared to the state-of-the-art CascadePSP, SegRefiner-Smai Small,, and SegRefiner Large, SAM consistently outperformed each on all four models.
YOLOvB YOLOud
Maak2Farmer
Image
Ground Truth
Initial Prediction
VOLOVe
Mask R-CNN
Mosk2Former SAM
CascadaPSP
SegRefiner
SAM
(a)
(b)
Figure 4: (a) Qualitative evaluation of SAM-Based refinement on YOLOv8 Nano, YOLOv8 X-Large, Mask R-CNN, and Mask2Former segmentations. (b) Qualitative comparison of CascadePSP, SegRefiner, and an optimized SAM prompt refining segmentation, producing an IoU of 0.960, 0.926, and 0.976, respectively.
Table 2: Paired (-Test of Refined vs Initial AP@50-95 95 by Model and Method on Powder Dataset
Model
Method
Mean AAP@50 95
Std AAP@50 95
95% CI
CascadePSP
33.09
0.0000
0.153
0.179
0.144 0.162
SegRefiner Small
5.84
<0.0001
0.037
0.219
0.025 0.050
YOLOv8 Nano
SegRefiner Large
7.60
<0,0001
0,052
0.231
0.038-0. 0.08
SAM (This Work)
42.38
<0,0001
0,203
0.186
0.194-0.28 0.213
CascadePSP
33.10
<0.0001
0.149
0.175
0.140 0.158
SegRefiner Small
6.89
<0.0001
0.042
0.208
0.030-0.054
YOLOv8 X-Large
SegRefiner Large
9.52
<0.0001
0.060
0.214
0.047-0.072
SAM (This Work)
41.19
<0.0001
0.201
0.190
0.192 0.211
CascadePSP
20.81
<0.0001
0.082
0.153
0.074 0.090
SegRefiner Small
11.02
<0.0001
0.058
0.205
0.048-0.069
Mask R-CNN
SegRefiner Large
12.37
<0.0001
0.068
0.215
0.058 0.079
SAM (This Work)
21.82
<0.0001
0.128
0.229
0.117-0.140
CascadePSP SegRefiner Small
16.50
<0.0001
0.059
0.140
0.052 0.067
4.95
<0.0001
0,020
0.154
0.012-0. 0,027
Fasm2Former
SegRefiner Large
10.36
<0.0001
0.036
0.137
0.030-0. 0.044
SAM (This Work)
19.61
<0.0001
0.081
0.161
0.073 0.089
10

---

Published in Transactions on Machine Learning Research (11/2025)
An ablation of these results, shown in Table highlights how incrementally adding each component continues to improve performance. Starting from single-point buffer, and then increasing to three points yields consistent gains across all four backbones. Switching from random to Voronoi placement provides an additional but smaller boost on three backbones, and a negligible decrease on the fourth (-0.0007). Lastly, introducing a bounding box led to the largest increase in performance, ranging from +0.0482 to +0.1248 increase on top of previous performance improvements achieved by the choice of the placement algorithm, the number of points, and the perimeter buffer.
baseline, adding 10% perimeter
Table 3: Prompt-construction ablation highlighting how segmentation refinement performance (mean AP@50-95) evolves as components are added. For each initial segmentation model, the columns report the mean and the change (A) from the prior step.
Ablation Step
YOLOvB Nano
YOLŌvS X-Large
Mask R-CNN
Mask2Former Mesn AP 95
Mean AP 95
A
Mean AP350 95
A
Mean AP350 95
Baseline:
Point
0. 0616
0.1999
0.1371
0.3973
10% perimeter buffer
0.4751
+0.0118
0.5369
-0.0370
0.4913
+0.0539
0.4055
+0.0082
Increase to 3 Points
0. 58211
+0.1090
0,5530
-0.0161
0.5390
+0.0676
0.4673
+0.0619
Vornnoi placement
0.5000
+000088
0.5987
-0.0457
0.5582
-0.0007
0.4686
40.0012
Add Bounding Box
0.7157
40.1248
0.0830
-0.0893
0.6497
+0.09155
0.5167
40.0482
Table 4: Performance comparison of refinement. methods across multiple initial model predictions for the Powder, General Microscopy, and Biological Cells from Scanning Electron Microscope Images. The best result in cach group is highlighted in bold.
Model
Powder
General Microscopy
Cells
AP
A AP
Aps5 musk
AP
AP
AP
A
A
A
AP
A Apsss
YOLO0d Neno
0.8979
0.6693
0.0806
0.5291
0.8950
0. 7866
0.2763
0.6526
0.5630
0.1927
0.0000
0.8519
Cascadel'PP
0.8450
0.7529
0.3781
0.6587
0,8910
0.7878
0.3427
0.6738
0.5061
0.4105
0.00012
0.3056
SegRefiter- Small
0.8078
0.6965
0.1843
0.5629
0,7174
0.6381
0.2099
0.5218
0.3375
0.1291
0.0000
0.1555
SegRefiner Large
0.8344
0.7267
0.3455
0.6355
0.7476
0.6784
0.3020
0.5760
0.5245
0,2924
0.0000
0.2723
SAM (This Work)
0.8812
0.8055
0.1591
0.7153
0.8930
0.8193
0.2300
0.6174
0.5922
0.12199
0.0016
0.3395
YOLOs8 X-Large
0,8352
0.6841
0.0964
0.5386
0.8817
9. 763633
0.2134
0.6195
0.5730
0.2152
060000
0.2624
CascadePSP
0.8116
0.7587
0,3870
0.6635
0,8851
0,7890
0.2761
0.6:198
0.5061
0.3986
0.0025
0.3021
SegReffner Sunall
0.8105
0.6865
0.1860
0.5610
0.70333
0.62115
(.2277
0.5172
0.4035
0,1250
0.0000
0.1762
SegRRelize- Large
0.8329
0.7375
0.3523
0.6409
0.7188
0.6375
0.2702
0.5/21
0.5307
0.1873
0.0016
0.2399
SAM (This Work)
0,8775
0.8102
0.4757
0.7211
0.8959
0,8132
0.2120
0.6404
0.5556
0.4038
0.0000
0.3198
Musk R-CNN
0.6876
0.5399
0.1061
0.4512
0.6653
0.5826
0.0502
0.4344
0,3438
0.1994
0.0000
0.1611
Cascadde'S
0.7254
0.6305
0.3245
0.5601
0.6870
0.5833
0.1775
0.4826
0.3519
0.1596
D.D010
0.1705
SegReliner- Small
0.6899
0.6050
0,2056
0.5001
0.6952
0.6196
0.2002
0.5070
0.2485
0.0761
0.0000
0.1082
SegRefiner Targe
0.7291
0.6434
0.3031
0.5586
0.7176
0.6317
0.2698
0.5397
0.3548
0.1754
0.0079
0.179A
SAM (This Work)
0.7788
0.7130
0.4692
0.6537
0.8959
0.8132
0. 0.2120
0.6-104
0.3289
0.1584
0.0000
0.1624
MaskiFormer
0.633%
(15111
0.1299
0.4925
0.8539
0, 757577
01.855
0.63999
6.3795
0.1873
0.0001
0.1890
CSP
0.6673
0.5805
0.2387
0.4955
0.8693
0.7337
0,2909
0.6313
0.3584
0,2258
0.0007
0.1950
+ SegRefiuer- Saaall
0.6300
0.4571
0.1176
(1.1116
0.6777
0.5752
01.8447
0.1792
0.26226
0.06311
D.D0R0O
0.1086
SegRellnee- Large
0,6666
0.3718
0.2357
0.4900
0.7340
0.6218
0.2676
0.5411
0.3706
0.1146
<0.0001
0.1617
SAM (This Work)
0.6548
0.5497
0.3132
0.5059
0.8588
0.7827
0.2003
0.6139
0.3533
0.2620
0.0006
0.2053
Extending this to additional AP thresholds, shown in Table 4 we found that SAM achieved the highest AP@95 across all four base segmentation models. At AP@50 and AP@75, SAM outperformed both SegRe- finer Small and SegRefiner Large on every initial sogmentation model, and exceeded CascadePSP on three of the four models. Expanding this analysis to the two additional datasets, general microscopy and cellular microscopy, as shown in Table ! we found that SAM continued to improve segmentation quality, and per- form comparably or better to the state-of-thh-art methods. For AP@75, scored the highest on all four initial segmentation models for both datasets. For AP@50, SAM scored the best in four of eight measurements, and second best in an additional two.
11

---

Published in Transactions on Machine Learning Research (11/2025)
Using a prompt with a bounding box; points, Voronoi placement, 10% perimeter buffer, and no coarse mask, SAM produced statistically significant improvement over the raw segmentations, and performed consistently better than or comparably to CascadePSP SegRe- finer across three microscopy datasets illustrating the generalizability of prompts for these scientific image datascts.
Key Takeaway 1:
5.1 Prompt Optimization
5.1.1 Impact of Bounding Boxes Prompt Performance
Of the 2,688 prompts tested, these were evenly split between no bounding box, as-detected bounding box, bounding box shrunk to 90% of its original size, and bounding box enlarged to 110% of its original size. To evaluate impact, paired I-lests were conducted, comparing each box configuration to an identical prompt without a box, as shown in Table 5 On average, including an as-detected bounding box had an average AP@50-95 improvement of 0.32. Shrinking this box to 90% of its size had a slightly larger improvement at 0,341, while enlarging the box had a slightly smaller improvement at 0.293.
Table 5: Paired Tests on Impact of Bounding Box Scale Segmentation Refinement (AP@50 95)
Box Scale
t
p
Mean AAP
Std AAP
95% CI
90%
105.77
<0.0001
0.341
0.167
0.334 0.347
100%
106.39
<0.0001
0.320
0.156
0.314-0.326
110%
103.99
<0.0001
0.293
0.146
0.287 0.298
5.1.2 Impact of Coarse Masks on Prompt Performance
Contrary to bounding boxes, including coarse masks in prompts consistently negatively affected prompt performance in our study. Broken down by mask augmentation, 90%, 100% (as-detocted), and 110%, paired tests revealed an average decrease -0.133 AP@50-95, as shown in Table Interestingly, mask augmentation found to have impact, with each having average decrease of -0.133, STD of 0.137, and 95% confidence interval of -0.138 -0.128. In the future, it would be beneficial to evaluate whether this trend persists on natural images, which may feature textured backgrounds, occlusions, and thin or elongated structures that could alter the utility coarse masks.
Table 6: Paired t-Tests on Impact of Coarse Mask Scale on Segmentation Refinement (AP@50 95)
Mask Scale
p
Mean AAP
Std AAP
95% Cl
90%
-50.38
<0.0001
-0.133
0.137
-0.138
-0.128
100%
-50.47
<0.0001
-0.133
0.137
-0.138
-0.128
110%
-50.38
<0.0001
-0.133
0.137
-0.138 -0.128
5.1.3 Combined Effect of Bounding Boxes and Coarse Masks
In addition to the independent effects of bounding boxes and coarse masks on performance, we found that their combined impact is even more pronounced. For example, prompts that include a bounding box out- perform those that omit it, while prompts that inchude a coarse mask underperform those that exclude Further, prompts featuring a box without a mask yield the highest average performance, whereas prompts with mask but no box yield the lowest, as shown in Figure 5A.
12

---

Published in Transactions on Machine Learning Research (11/2025)
Mask Inclusion Omission
Distortion with No Mask
0.8
0.6
AP0050-95
0.4
0.4
02
0.1
0.0
Box & Mask
No Mork
Box Mask
No No Masik
0.9
1.
14
Bounding Distortion
Figure 5: Visualizing impact of bounding boxes and masks on AP@50-95 when using SAM-baser refinement.
Key Takeaway 2: Prompts that included a bounding box improved AP@50 by an average of 0.32 comparod identical prompts without box, whereas prompts that inchuded coarse mask decreased AP@50 95 by an average of 0.133. Overall, the highest performance was achieved by prompts that included bounding box and excluded coarse mask.
Focusing on the best of these combinations (yes box and no mask), Figure 5 highlights the distribution of AP@50-95 scores when the bounding box size shrunk to 90%, enlarged to 110% of its original size. Minor differences in distributions can be observed, such as a slightly lower tail distribution for prompts with an enlarged box (0.560 compared to 0.571) and a slightly smaller distribution when the shrunk bounding boxes (0.175 compared to 0.209). However, we conclude that overall box augmentation had no significant effects on performance.
5.2 POI Placement Impact
Analyzing the effect of POI prompt placement, we considered the individual and combined effect of the number of points (k), POI placement algorithm (ø), and perimeter buffer (y). In addition, evaluated the effect of these parameters in the presence and absence of a bounding box, allowing to quantify how box inclusion modulates the influence of POI placement stratogics segmentation refinement.
5.2.1 Algorithm Selection (ø)
To analyze the impact of alternate PO1 placement algorithms, we treated random placement as the baseline, and conducted paired (-tests, measuring the change in AP@50-95 as a result of switching to distance maxi- mization or Voronoi placement. For this analysis, we performed two sets of paired t-tests: one over the subset of prompts without bounding box or mask (Table 7 and another over the subset of all prompts with a bounding box but no mask (Table Sh. As shown in Table Voronoi performed the best when no box included, with mean AP@50-95 improvement of 0.007 over random. In contrast, distance maximization performed the worst with average decrease in performance of -0.146. In tests prompts including bounding box, the mean AP@50-95 increased significantly for all three algorithms. Random incrcased from 0.541 to 0.645, Distance Max increased from 0.392 to 0.622, and Voronoi increased from 0.548 to 0.644. Additionally, when a box was included, as shown Table S the effect the placement algorithm dimin- ished. For example, the performance decrease due using Distance Max narrowed from -0.146 -0.023. Similarly, the previously statistically significant improvement of 0.007 AP@50-95 for Voronoi over Random (p < 0.0001) became non-significant decrease of -0.001 (p 0.1657).
13

---

Published in Transactions on Machine Learning Research (11/2025)
Table 7: Paired teTsss Comparing Placement Algorithms (No Box, No Mask)
Comparison
t
Mean AAP
Std AAP
95% C1
- Max
-20.17
<0.0001
-0.146 0.007
0.108 0.020
-0.160
0.132
Random Voronoo
5.30
<0.0001
0.005-0.010
Table 8: Paired t-Tests Comparing Placement Algorithms (Box, No Mask)
Comparison
t
Mean AAP
Std AAP
95% CI
Random Distance Max
-18.93 Random + Voronoi -1.39
<0.0001
-0.023
0.031
-0.025
0.021
0.1657
-0.001
0.012
-0.002 0.000
Key Takeaway 3: In this study, the placement of POIs along the boundary of an initial detection performed significantly worse than random or Voronoi placement, particularly when no bounding box was provided. However, when a bounding box was provided, the gap between algorithm selections was substantially reduced.
5.2.2 Number of Points (l:)
Depending on the algorithm and the number points (k), we note that points could placed in drastically different ways, as depicted in Figure To test this effect, specifically related to the number of points, we separated prompts by placement algorithm and box inclusion. Without bounding box, adding more POIs yielded clear gains for random (up to +0.091 AP@50-90 95, p 0.0001) and Voronoi (up to +0.083 AP@50-90 95, 0.0001), while Distance Max steadily lost performance (down to -0.105, 0.0001), as shown in Table 9 However, when a bounding box was included, these trends were reversed. Random and Voronoi went from a steady increase to a slight decrease in performance, and Distance Max's decrease in performance was reduced from -0.105 to -0.044.
Point
2 Point
3 Point
4 Point
Point
Random
Distance Maximization
Voronoi
Figure 6: Sample placement of points using Random, Distance Maximization, and Voronoi placement from k to k= = 5 points.
Key Takeaway 4: In this study, including additional POTs improved performance for random and Voronoi placement, but decreased performance for Distance Max. However, these effects were reduced once a box included.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Table 9: Paired f-Tests Increasing POL Count (1 vs k) on AP@50 95 (No Box, No Mask)
Algorithm
Comparison
t
p
Mean AAP
Std AAP
95% C1
1-22
7.13
<0.0001
0.035
0.027
0.025 0.044
1-+3
14.76
<0.0001
0.063
0.024
0.055-0.071
1-+4
18.49
<0.0001
0.079
0.024
0.070-0.087
Random
1-+5
19.93
<0.0001
0.089
0.025
0.081-0.098
1-66
21.66
<0.0001
0.091
0.024
1-77
16.40
<0.0001
0.090
0.031
0.079 0.100
1-22
-4.49
0.0001
-0.052
0.065
-0.077
0.032
1-3
-6.06
<0.0001
-0.082
0.076
-0.111-0 0.059
1-4
-6.77
<0.0001
-0.093
0.078
-0.121
0.069
Distance Max
1-55
-6.70
<0.0001
-0.098
0.083
-0.128
0.073
1-6
-6.86
<0.0001
-0.103
0.085
-0.134
0.077
1-77
-7.02
<0.0001
-0.105
0.085
-0.137
0.079
1-2
10.40
<0.0001
0.022
0.012
0.018 0.026
1-33
16.12
<0.0001
0.053
0.019
0.047 0.059
1-44
20.95
<0.0001
0.072
0.019
0.065-0.078
Voronoi
1-55
17.65
<0.0001
0.077
0.025
1-+6
19.71
<0.0001
0.082
0.024
0.074 0.090
1-+7
25.98
<0.0001
0.083
0.018
0.076-0.089
Table 10: Paired t-Tests of Increasing POI Count (1 vs k) on AP@50-95 (Box Only, No Mask)
Algorithm
Comparison
p
Mean AAP
Std AAP
95% CI
1-32
-2.11
0.0372
-0.003
0.012
-0.005 0.000
1-33
-3.53
0.0006
-0.004
0.012
-0.007-0. 0.002
1-44
-8.03
<0.0001
-0.009
0.011
-0.011-0.007
Random
1-55
-8.85
<0.0001
-0.011
0.012
-0.013 0.008
1-66
-10.40
<0.0001
-0.122
0.012
-0.015
0.010
1-x7
-10.58
<0.0001
0.013
0.012
0.016-0.011
1-2
-7.84
<0.0001
0.014
0.018
-0.018 0.011
1-33
10.66
<0.0001
-0.021
0.020
-0.025 0.018
1-44
-9.26
<0.0001
-0.044
0.036
-0.041-0.00
Distance Max
1-5
-10.30
<0.0001
-0.422
0.039
-0.050
0.034
1-6
-12.38
<0.0001
-0.455
0.036
-0.052 0.038
1-77
-13.11
<0.0001
-0.044
0.033
0.051-0.085
1-2
-0.73
0.4689
-0.001
0.011
-0.003 0.001
1-3
-5.40
<0.0001
-0.055
0.009
0.007-0.003
1-44
-4.56
<0.0001
-0.006
0.014
Voronoi
1-55
-8.91
<0.0001
-0.009
0.010
-0.011
0.007
1
-9.77
<0.0001
-0.013
0.013
-0.06 0.010
1-77
-00077
<0.0001
-0.013
0.013
-0.015 0.010
5.2.3 Perimeter Buffer (7)
Based on the theory that POIs closer to the edge have a higher chance of being outside the ground truth, and therefore degrade performance, we implemented a perimeter buffer where POIs could not be placod. To evaluate this, we tested a 0% buffer (no buffer) and 5%, 10%, and 15% buffer. When no box was included, random placement had significant change at. 5% buffer (+0.002 AP@50-95, 0.3854),
15

---

Published in Transactions on Machine Learning Research (11/2025)
and small improvement 10% (+0.008, 0.0017) and 15% (+0.012, p 0.0002). Similarly, Voronoi placement had no significant change at a 5% buffer (+0.004, p 0.0654), and small improvement at 10% (+0.007, p 0.007) and 15% (+0.006, = 0.0064). However, Distance Max had much larger and more statistically significant improvement at 5% (+0.053, p < 0.0001), 10% (+0.093, < 0.0001), and 15% (+0.093, p < 0.0001). When evaluating prompts with a bounding box, we observed similar trend reversal that was found the placement algorithm selection and the number of POI selections. Random and Voronoi had no statistically significant change in AP@50-95. The improvements for Distance Max dropped from +0.053 to +0.007 at 5%, from +0.093 to +0.006 at 10%, and from +0.093 to -0.024 at 15%.
Table 11: Effect of Perimeter Buffer on AP@50 95 (No Box, No Mask)
Algorithm
Buffer
t
Mean AAP
Std AAP
95% C1
0%-5%
0.87
0.3854
0.002
0.021
-0.003 0.008
Random
0%-110%
3.30
0.0017
0.008
0.019
0.003 0.013
0%-+15%
4.02
0.0002
0.012
0.022
0.006 0.017
0%-55%
5.27
<0.0001
0.053
0.076
0.035 0.073
Distance Max
0%-+10%
6.54
<000001
0.093
0.107
0.066-0.122
0%-115%
5.32
<0.0001
0.093
0.131
0.060-0.128
0%-55%
1.89
0.0645
0.004
0.017
-0.000-0.009
Voronoi
-0%
2.84
0.0063
0.007
0.018
0.002-0.011
0% 15%
2.84
0.0064
0.006
0.017
1.002-0.01
Table 12: Effect of Perimeter Buffer AP@50 95 (Box Only, No Mask)
Algorithm
Buffer
t
P
Mean AAP
Std AAP
95% CI
0%-+5%
1.60
0.1118
0.001
0.011
-0.000 0.003
Random
0%-+10%
1.46
0.1475
0.002
0.014
-0.001-0.004
0%-+15%
2.55
0.0118
0.002
0.012
0.001-0.004
0%-+5%
4.71
<0.0001
0.007
0.018
0.004-0.009
Distance Max
00%+
4.16
0.0001
0.006
0.020
0.003 0.009
0%-+15%
-8.04
<0.0001
-0.025
0.040
-0.031 0.019
0%-+5%
-1.33
0.1869
-0.001
0.011
-0.003-0.000
Voronoi
10%%
0.86
0.3912
0.001
0.010
-0.001 0.002
0%-+15%
0.42
0.6772
0.000
0.011
1..00-0.002
Key Takeaway 5: When no bounding box is provided, introducing a small perimeter buffer (10-15%) around the mask edge improved AP@50-95 scores, particularly for Distance Max.
Connecting the Dots. To visualize the interconnected nature of these parameters, Figure 7 highlights the effect of placement algorithm (ø), number of points (k), perimeter buffer (y). box inclusion across the four initial segmentation models. Each row shows one placement strategy (Distance Max top, Random in the middle, Voronoi bottom), and each column corresponds to initial segmentation model (YOLOv& Nano, YOLOv8 X-Large, Mask R-CNN, Mask2Former, left to right). Within each graph, increase along the X-axis indicates a growing perimeter buffer (7), while an increase along the Y-axis indicates an increase in reported AP@50-95. Each colored line represents the number POIs used (k), with solid lines indicating a bounding box was included and dashed lines indicating the bounding box was excluded.
In these results, we confirm previous findings that bounding boxes improved results with every single prompt. performing better with bounding box than its counterpart without, bounding box.
16

---

Published in Transactions on Machine Learning Research (11/2025)
YO: Nano Distance Max
101 OvB X-large Distance Max
Mask R-CNN Distance Max
MaskeFomeer Distance Nax
0.2
YOLOUB Nono Random
VOLOvS Random
Maak R.CNN- Randem
Forka Random
50050000
X-Large
0.6
Ah930-05
35
32
22
34
10
14
15 12
POOss (Box)
POtss [Box]
POIS (Box)
POIs (Box) 4 POIs (No Box)
POIs (Box)
(Box) 6 POl (No Box)
POls (Box)
1 POSS (No Box]
2 POrs (No Box)
POIS (No Box)
5 PPt (No Box)
7 PPtt (No Box)
Figure 7: Evaluation of bounding box inclusion, POI placement algorithms, number of POIs, and perimeter buffer across multiple instance segmentation architectures.
5.3 Iterative Refinement
To evaluate whether any prompt could further improve an already refined mask, we implemented a two- stage protocol. In Stage 1, for each base segmentation model (see Section 4.2 and Table D. we applied each prompt configurations {Pt} to the raw model outputs and selected the top-performing prompt Pt, In Stage 2, we first used P to generate once-refined masks under optimal prompting conditions, then re-ran SAM with each prompt P, on those once-refined masks to produce twice-refined outputs. We measured AP@50- -5 for every P, in Stage 2 and compared it to the corresponding Stage 1 score to see if any prompt yielded additional gains.
As shown in Figure A, the range of twice-refined AP@50-95 scores was much narrower than once-refined (Fig. 5), particularly for prompts including a bounding box, or no bounding box and no coarse mask. For example, the range of once-refined AP@50-95 scores for prompts containing a box and mask decreased from 0.39 to 0.18, prompts containing a box and mask decreased from 0.21 to 0.10, and prompts containing neither box nor mask dropped from 0.58 to 0.22. While we saw significant reduction in the spread of recorded values, this was from a rising tail value, and the maximum recorded value actually decreased from once-refined to twice-refined segmentations for each group. In fact, for all (P:). 59.24% achieved higher AP@50-95 when run on the once-refined P masks compared to the raw outputs. However, none of the twice-refined outputs from any P; produced a closer segmentation to the ground truth than the once-refined mask produced by P'.
To analyze this further, paired t-tests comparing once-refined and twice-refined AP@50 95 were conducted, shown in Table I For improved clarity, prompts were divided into five groups according to their once- refined AP@50-95 percentiles: 0-20%, 20-40%. 40-60%, 60-80%, and 80-100%. Prompts in the 0-20% group AP@50-95, p < 0.0001). As the percentile group increased, the experienced the largest improvement (+0.034 A
improvement decreased. For example, 20-40% had a smaller improvement (+0.008 AP@50-95, p < 0.0001), and 40-60% had an even smaller improvement (+0.003 AP@50-95, p < 0.0001). Prompts in the 60-80% range had an average improvement of 0.000 with no statistical significance (p = 0.4481), and prompts within the upper 80th percentile had statistically significant. decrease in performance (-0.006 AP@50-95, p < 0.0001).
17

---

Published in Transactions on Machine Learning Research (11/2025)
Mask Inclusion Omission
Distortion Mask
0.6
0.5
AP 05.05
0.4
0.4
02
0.1
0.1
0.0
Box Mask
Box No Mark
Dox Klsk
No No Mask
0.9
Bounding Distortion
Figure 8: Impact of Box and Mask Prompts on Secondary Refinement
Table 13: Paired t-Test of Onco-Refined vs Twice Refined Segmentations
Percentile
Mean AAP
Std AAP
95% C1
0%-20%
19.05
<0.0001
0.034
0.059
0.031
0.038
40%%40%
12.23
<0.0001
0.008
0.022
0.007
0.010
40% 60%
5.89
<0.0001
0.003
0.016
0.002
0.004
60%-80%
0.76
0.4481
0.000
0.015
-0.001
0.001
80% 100%
-14.00
<0.0001
-0.006
0.015
-0.007
-0.006
Key Takeaway 6: Prompts that initially performed quite poorly did improve when using a once- refined segmentation. However, no twice-refined segmentation outperformed the highest-scoring once- refined segnentation, and high-performing prompts consistently performed worse twico-refined than once-refined.
5.4 Evaluating Generalizability
While this work provides exlensive evaluation of prompt construction of SAM scientific images, we also tested its generalizability under two alternate conditions. First, we tested if our empirically deter- mined optimal prompt construction still yielded similar performance SAM2, with SAM2 newer SAM architecture designed for videos that is meant to continue to also provide image-segmentation capabilities. Further, we also tested this same prompt using SAM in a domain distinct from scientific images to evaluate generalizability.
5.4.1 Alternative SAM Architectures: SAM vs. SAM2
SAM2 released in 2024 Ravi et al. 20244 one year after SAM Kirillov et al. 2023 was designed to not only handle the segmentation of still images but now also of videos. In this study, we applied the same prompt points with Voronoi placement, 10% perimeter buffer, and bounding box) on all four SAM2 variants (base+, tiny, small, and large), shown in Table 14 Compared to the unrefined mask, each architecture improved performance with AP@50 increasing from 0.8469 to 0.8717-0.8745 AP@75 increasing from 0.6768 to 0.7521-0.7660, AP incrcasing from 0.0912 to 0.3212-0.3672, and AP@50-95 increasing from 0.5383 to 0.6488-0.6670. While improving over the unrefined baseline, all SAM2 variants showed slightly lower performance than SAM on single-image refinement, with the gap most pronounced at stricter thresholds.
For example, SAM2-Large reached AP@50 of 0.8722 versus 0.8806 for SAM, but at AP@95 obtained 0.3339 versus 0.4712. This slight decrease in performance consistent with previous findings in the literature that SAM2 is not good at fine-grained detail for images Pei al. 2024
18

---

Published in Transactions on Machine Learning Research (11/2025)
Comparing SAM to SAM2 segmentation refinement performance on the powder dataset using 3 points, Voronoi placement, 10% perimeter buffer, and a bounding box.
Table 14:
Method
A
AP@75
AP@95
AP@50-95
Unrefined (initial)
0.8469
0.6768
0.0912
0.5383
SAM2-Basc+
0.8717
0.7621
0.3672
0.6670
SAM2-Tiny
0.8748
0.7660
0.3287
0.6565
SAM22-Sall
0.8731
0.7521
0.3213
0.6488
SAM2-Large
0.8722
0.7656
0.3339
0.6572
SAM (This Work)
0.8806
0.7954
0,4712
0.7157
5.4.2 Alternative Image Domains
To test generalizability, we evaluated our recommended prompt configuration on natural-scene datasets contrast of target scientific image datasets. For this, selected ECSSD (Shi al. 2015), collection
of 1,000 natural images with textured backgrounds, specifically designed to increase segmentation difficulty. Using three initial segmentation models (RMBG-2 (BRIA AI 2025). BiReeNet-DIS (Zheng et al. 2024).
and RMBG-1.4 (BRIA Al 20024), we compared the intersection-vve--miioo (IoU) of refined segmentations to their initial segmentation. Averaged across the three models, SAM (with a prompt of points, Voronoi placement, a 10% perimeter buffer, and bounding box) improved IoU by +1.79, as shown in Table 15 comparison, CascadePSP produced an average ToU gain +0.62, while SegRefiner-.RR and SegRefiner-HHe both degraded performance. These results on natural-scene dataset indicate that the proposed prompt configuration can transfer beyond microscopy and functions gencral, model-agnostic refiner.
Table 15: Average ToU across three initial segmentation models, RMBG-2 BRIA 2025
BiRefNet (Zheng
et al. 2024 and RMBG-1.4 (BRIA Al
2024) on the ECSSD dataset (Shi et al. 2015).
Method
RMBG-2 BiRefNet-DIS RMBG-1.4
Mean
Unrefined (initial)
86.85
79.30
78.09
81.41
CascadePSP
86.39
81.09
78.61
82.03
SegRefiner LR
70.25
67.27
65.31
67.61
Sog
68.50
66.40
64.39
66.43
SAM (This Work)
87.78
83.05
78.77
83.20
5.5 Limitations
While this study demonstrates statistically significant results showing that SAM perform similarly to better than state-of-the-att segmentation refinement techniques, it does have few limitations. A primary limitation is the dependence the quality and completeness of the initial masks. Because SAM's refinement leverages the spatial support and appearance cues present in the initial segmentation, large misalignments or missing instances limit its ability to effectively refine object boundaries. This highlighted in Figure g where the same prompts are used refine the same image, except. that we use initial segmentations from YOLOv8 Nano (with 76 detccted objects) and Mask2Former (with 64 detected objects). SAM successfully refines many of the detected objects, improving AP@50 95 from 0.8048 to 0.9307 for YOLOv8 Nano and from 0.6560 to 0.7787 for Mask2Former. However, despite using the same prompt construction, the same SAM architecture, and the same image, the refined Mask2Former result is significantly lower. This occurs because SAM is strictly a refiner in this setting and has no way to recover missed segmentations (marked with a red X).
Second, SAM's performance is sensitive "false POIs (points placed outside the true object), which can rapidly deteriorate mask refinement. We mitigate this through perimeter buffer, preventing POIs from being placed near the segmentation boundary, where they most likely to be incorrect. However, it remains
19

---

Published in Transactions on Machine Learning Research (11/2025)
YOLOvB Nano
Mask2Former
Unrefined
Refined
Unrefined
Refined
Count: 76- AP50-95: 0.8048
Count: 76 AP50-95: 0.9307
Count: 64 AP50095: 0.6560
Count: 64 A450099995 0.7787
Figure 9: Example of SAM's dependence on initial segmentations. YOLOv8 Nano (left) and Mask2Former (right) on the same image show increased performance after refinement, but missed instances in the initial masks (marked with red Xs) cannot be recovered.
to be explored if this may generalize to more complex shapes. A more robust approach may intelligently select or augment POI locations. This effect is highlighted in Figure I0 Using distance max with no perimeter buffer, where many points could be placed in over-segmented false-positive regions, many of the particles had blurred edges, including the object and its surrounding area, even "halo' highlighting the surrounding area but not the object, resulting in a decrease of AP@50-95 from 0.6576 to 0.2633. However, by introducing perimeter buffer of 10%, the likelihood of false-points being placed significantly decreases, resulting in cleaner segmentations, and an AP@50-95 increase from 0.6576 to 0.8265.
Unrefined
Ground Truth
Distance Max 0% Buffer
Distance Max 10% Buffer
Count: 84 AP50-95: 0.6576
Count: 88
Count: AP50-95: 0.2633
Count: APS-0509 0.8265
Example of SAM's sensitivity to false POIs, particularly for Distance-Max point placement. With no perimeter buffer, performance decreases from 0.6576 to 0.2633, but with a 10% buffer, performance increases from 0.6576 to 0.8265.
Figure 10:
Third, SAM is most reliable when object boundaries well defined. In images with low-contrast gradually varying boundaries, refinement can leak across neighboring instances or into the background. As shown in Figure I1 where cells exhibit gradual border transitions, SAM blends several annotations in the bottom right corner, which did not occur for CascadePSP or SegRefiner. For this reason, SAM was not as consistently outperforming CascadePSP and SegRefiner for the additional datasets on all metrics. Despite this, SAM still improved over the unrefined masks and exceeded CascadePSP or SegRefiner on some metrics, such AP075.
Fourth, work is focused scientific images, and more specifically, microscopy images. Given SAM's domain-dopendent performance, future works should evaluate the generalizability of these prompt- augmentation strategies on additional domains. In support of this, we have released our reusable code, enabling researchers to rapidly extend these results for their target domains of interest.
Lastly, although we comprchensively explored 2,688 prompt configurations across four base segmentation models (10,752 total evaluations), our search was not exhaustive. There possibility that alternative prompting strategies may yield further improvements.
20

---

Published in Transactions on Machine Learning Research (11/2025)
Raw Image
Ground Truth
Predicted
CascadePSP
SegRefiner
SAM
Figure 11: Qualitative example of SAM's limitations for segmenting objects with gradual boundarics, such as the cells in the lower right corner, where multiple objects are blended together.
6 Conclusion
Accurate object detection and segmentation are necessary for numerous tasks ranging from high-throughput object quantification autonomous driving. Recently, there has been growing attention modcl-agnostic segmentation refinement methods and post-processing detections for improved performance. In this work, we provide statistically significant evidence that SAM can perform comparably or better than state-of-the- refinement techniques, like CascadePSP and SegRefiner, by leveraging an appropriate visual prompt config- uration. However, SAM's performance depends on the prompts it is given. Recent works have conducted a preliminary analysis on the effect of prompting, but are limited in that they frequently evaluate fewer than ten prompt combinations (Cheng et al. 2023a Hu et al. 2023 Mayladan et al. 2023). They have also rarely studied the combinod effect of more than one prompt type together (c.g., points, boxes, and masks together) [Cheng el. al. 2023a Dai et al. 2023) In this work, provide the first large-scale analysis, including 2,688 prompt combinations across four initial segmentation models, resulting in total of 10,752 evaluations.
Through this experimentation, we uncover several key takcaways that lead to guidelines for prompt design, improving segmentation performance for scientific applications, summarized here:
Prompts including a bounding box have an improvement 0.320 AP@50-95 over identical prompts without a box across the 10,752 evaluations on microscopy datasets.
Prompts including a coarse mask have a decrease in performance of -0.133 AP@50-95 over identical prompts without mask.
When no bounding box is provided, placing POIs along the boundary of initial segmentations is detrimental to performance, but can be mitigated by perimeter buffer, or POT placement algorithm, such as Voronoi, that prioritizes more central regions.
Iterative refinement was not found to yield improved performance over the highest-scoring refined mask.
These results provide a statistically significant evaluation of how prompt-augmentation can improve segmen- tation performance. Moving forward, this research should be extended to study other domains of interest, further verifying its generalizability. Future work can also focus on developing additional solutions for POI placement, which may mitigate the challenge of "false POIs' placed outside the ground truth.
7 Broader Impacts
This work poses no direct ethical or socictal risks, as it operates in data-agnostic, model-agnostic setting locused strictly on segmentation refinement. However, the underlying Segment Anything Model (SAM) is energy-inte energy-intensive: its pretraining required 68 hours on 256 A100 GPUs, and each refinement pass is heavier than a singlo-stage segmenter. To limit inferenco-time carbon footprint, we recommend (1) reserving SAM- based refinement for cases where pixel-level boundary accuracy essential, and (2) reusing SAM's large encoder embeddings across multiple objects rather than re-computing them for each mask.
21

---

Published in Transactions on Machine Learning Research (11/2025)
8 Code Availability
All code and refinement results used for analysis in this study available at: https://github.coom @pr.cen14//aaaterrrgnaaaaammopp
9 Acknowledgments
This research was supported part by the National Science Foundation under Grant NRT-HDR-2021S77 and in part by the National Science Foundation Graduate Research Fellowship Grant 2024369094. We thank Dr. Kyle Tsaknopoulos and Dr. Kiran Judd for their help with data collection and their insights into materials characterization, well the DAISY Research Lab and the Cote Research Lab for their feedback and support.
References
Dosovitskiy Alexey. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint Xixi:: 2010.11929 2020.
BRIA Al. Rmbg-1.4: Bria background removal v1.4. https///ungiigffe..coo/o//i///// 2024. Hugging Face model card; License: bria-rmbg-1.4.
BRIA AI Rmbg-2-maas 2-matting (bria background removal v2.0). MMBCC2-MMttting 2025. Hugging Face model card; License: bria-rmbg-2.0.
http:://aughaggaaccc////11111
Keyan Chen, Chenyang Liu, Hao Chen, Haotian Zhang, Wenyuan Li, Zhengxia Zou, and Zhenwei Shi. Rsprompter: Learning to prompt for remote sensing instance segmentation based on visual foundation model. IEEE Transactions on Geoscience and Remote Sensing, 2024.
Bowen Cheng, Ishan Misra, Alexander Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-atiention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 1290 1299, 2022.
Dongjie Cheng, Ziyuan Qin, Zekun Jiang, Shaoting Zhang, Qicheng Lao, and Kang Li. Sam medical images: A comprehensive study on three prompt modes. arXiv preprint ar arxi::2305. 00035, 2023a.
Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, and Chi-Keung Tang. Cascadepsp: Toward class-agnostic and very high-resolution segmentation via global and local refinement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8890-8899, 2020.
Junlong Cheng, Jin Ye, Zhongying Deng, Jianpin Chen, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, et al. Sam-med2d. ar preprint arXiv:2308. 16184, 2023b.
Ryan Cohn, Iver Anderson, Tim Prost, Jordan Tiarks, Emma White, and Elizabeth Holm. Instance seg- mentation for direct measurements of satellites in metal powders and automated microstructural charac- terization from image data. Jom, 73(7):2159 2172, 2021.
Haixing Dai, Chong Ma, Zhiling Yan, Zhengliang Liu, Enze Shi, Yiwei Li, Peng Shu, Xiaozheng Wei, Lin Zhao, Zihao Wu, et al. Samaug: Point prompt augmentation for segment anything model. arXio preprint arXiv:2307.01182 2023.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1961-2968 2017.
Chuanfei Hu, Tianyi Xia, Shenghong Ju, and Xinde Li. When sam meets medical images: An investigation of segment anything model (sam) on multi-phase liver tumor segmentation. arXio preprint or 2Xiv:2304.08500 2023.
22

---

Published in Transactions on Machine Learning Research (11/2025)
Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang. Mask scoring r-cnn.
In Proceedings of the IEEE/CVE conference. on compuler vision and pallern recognilion, pp- 1409-6415 2019.
Wei Ji, Jingjing Li, Qi Bi, Tingwei Liu, Wenbo Li, and Li Cheng. Segment anything is not always perfect: An investigation of sam on different real-world applications, 2024.
Idan Kassis, Dror Lederman, Gal Ben-Arie, Maia Giladi Rosenthal, Iaan Shelef, and Yaniv Zigel. Detection of breast cancer in digital breast tomosynthesis with vision transformers. Scientific Reports, 14(1):22149, 2024.
Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu, et al. Segment anything in high quality. Advances in Neural Information Processing Systems, 36, 2024.
Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 9809, 2020.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tote Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015 1056-026,44000 2023.
Quang H Le. GAN mask R-CNN: inslance semantic segmentation benefils from generalive adversarial networks. PhD thesis, Massachusetts Institute of Technology, 2020.
Chengyin Li, Rafi Ibn Sultan, Prashant Khanduri, Yao Qiang, Chetty Indrin, and Dongxiao Zhu, Au- toprosam: Automated prompting sam for 3d multi-organ segmentation. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5570-3580. IEEE, 2025.
Yuqi Lin, Hengjia Li, Wenqi Shao, Zheng Yang, Jun Zhao, Xiaofei He, Ping Luo, and Kaipeng Zhang, SAM- Refiner: Taming segment anything model for universal mask refinement. In The Thirteenth International Conference Learning Representations, 2025. URL https: wttp:/areeeeeeeeeeeoooorrrrtdeed
Jun Ma, Yuting He, Feifci Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024.
Ali Mayladan, Hasan Nasrallah, Hasan Moughnieh, Mustafa Shukor, and Ali Ghandour. Zero-shot refine- ment of buildings' segmentation models using sam. ar preprint ar wxxx:2310.018//3 2023.
Zhe Min, Jiewen Lai, and Hongliang Ren. Innovating robot-assisted surgery through large vision models. Nature Reviews Electrical Engineering, pp. 1 14, 2025.
Jialun Pei, Zhangjun Zhou, and Tiantian Zhang. Evaluation study sam 2 for class-agnostic instance-level segmentation. arXiv preprint arxxu:2409. 0276677 2024.
Stephen Price, Kiran Judd, Kyle Tsaknopoulos, Rodica Neamtu, and Danielle L Cote. Dualsight: multi-stage instance segmentation framework for improved precision. Scientific Reports, 15(1):27521, 2025.
Stephen E Price, Matthew A Gleason, Bryer C Sousa, Danielle L Cote, and Rodica Neantu. Automated and refined application of convolutional neural network modeling to metallic powder particle satellite detection. Integrating Malerials and Manufachuring Innovation, 10(4):661 676, 2021.
Jorge Quesada, Mohammad Alotaibi, Mohit Prabhushankar, and Ghassan AlRegib. Pointprompt: A multi- modal prompting dalasel for segment anything model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1004-160,,1111 2024.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chailanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam Segment anything in images and videos. arxiv preprint crxi:22408.000 00714, 2024. URL https::/ //aaxivv.. //taaaii..gg.aaa...../. $408.00711
23

---

Published in Transactions on Machine Learning Research (11/2025)
Matan Rusanovsky, Ofer Beeri, and Gal Oren. An end-to-end computer vision methodology for quantitative metallography. Scientific. Reports, 12(1) (2(1):4776 2022.
Chuyun Shen, Wenhao Li, Yuhang Shi, and Xiangfeng Wang. Interactive 3d medical image segmentation with sam 2. arXiv preprint arXiv:2408.0066 02635, 2024.
Jianping Shi, Qiong Yan, Li Xu, and Jiaya Jia. Hierarchical image saliency detection on extended essd. IEEE transactions on pattern analysis and machine intelligence, 38(4)7117779,, 2015.
Milad Sikaroudi, Maryam Hosseini, Ricardo Gonzalez, Shahryar Rahnamayan, and HR Tizhoosh. General- ization of vision pre-trained models for histopathology. Scientifie reports, 13(1);6065, 2023.
Chufeng Tang, Hang Chen, Xiao Li, Jiammin Li, Zhaoxiang Zhang, and Xiaolin Hu. Look closer to segment better: Boundary patch refinement for instance segmentation. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognition, pp. -39335, 2021.
Kai Tang, Yuguo Pei, Xiaobo Wang, and Leilei Qu. Improvement mask r-cnn algorithm for ore segmen- tation. Electronics, 14(10):2025, 2025.
Alex Tonncau. General microscopy dataset, apr 2023. URL http:://aalerre..oobollo.... parciellrreaaddd///srrr--------- visited on 2025-06-12.
Ultralyties, Yolov8 architecture explained: Exploring the yolov8 architecture, 2023. URL https://yolove:
org Accessed: 2024-07-07.
Mengyu Wang, Henghui Ding, Jun Hao Liew, Jiajun Liu, Yao Zhao, and Yunchao Wei. Segrefiner: Towards model-agnostic segmentation refinement with discrete diffusion process. Advances in Neurol Information Processing Systems, 36, 2024.
NP Wankhade, VP Sale, RS Yadav, PC Jikar, SR Gadgekar, and NB Dhokey. Metallurgical microstructure classification using enn: A comprehensive study on heat treatment analysis for steel. Materials Today; Proceedings, 2024.
Zhaoyang Wei, Pengfei Chen, Xuehui Yu, Guorong Li, Jianbin Jiao, and Zhenjun Han. Semantic-aware sam for point-prompei instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3585 3594, 2024.
Junde Wu, Ziyue Wang, Mingxuan Hong, Wei Ji, Huazhu Fu, Yanwu Xu, Min Xu, and Yueming Jin. Medical sam adapter: Adapting segment anything model for medical image segmentation. Medical image onalysis, 102:103547, 2025.
Hiroaki Yamagiwa, Yusuke Takase, Hiroyuki Kambe, and Ryosuke Nakamoto. Zero-shot edge detection with scesame: Spectral clustering-basee ensemble for segment anything model estimation. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 541-551, 2024.
Chongkai Yu, Anqi Li, Xiaochao Qu, Luoqi Liu, and Ting Liu. Sam-ref: Rethinking image- prompl synergy for refinement in segment anything. arXiv preprint 01855;2008.11555, 2024.
Yuki. Cellular microscopy dataset, 2024. URL cell-iv3b1 visited on 2025-06-12.
www.:/oow...rrroooooooooooo. ngldl/
Chaoning Zhang, Fachrina Dewi Puspitasari, Sheng Zheng, Chenghao Li, Yu Qiao, Taegoo Kang, Xinru Shan, Chenshuang Zhang, Caiyan Qin, Francois Rameau, et al. A survey on segment anything model (sam): Vision foundation model meets prompt enginecring. arXiv preprint arXiw:2306006211 2024a.
Chunhui Zhang, Liu, Yawen Cui, Guanjic Huang, Weilin Lin, Yiqian Yang, and Yuchong Hu. A com- prehensive survey on segment anything model for vision and beyond. arXiv preprint orXiv:2305.08190 081966 2023a.
24

---

Published in Transactions on Machine Learning Research (11/2025)
Dingyuan Zhang, Dingkang Liang, Hongcheng Yang, Zhikang Zou, Xiaoqing Ye, Zhe Liu, and Xiang Bai. Sam3d: Zero-shot 3d object detection via segment. anything model. arXiv preprint or orXiu:2306. 02245. 2023b.
Gang Zhang, Xin Lu, Jingru Tan, Jianmin Li, Zhaoxiang Zhang, Quanquan Ti, and Xiaolin Hu. Refinemask: Towards high-quality instance segmentation with fine-grained features. In Proceedings of the IEEE/CVE conference compuler vision and pallern recognilion, pp. 6861-6869, 2021.
Sheng Zhang, Kaiyu Wang, Huajun Zhang, Tong Wang, Xiguang Gao, Yingdong Song, and Fang Wang. An improved yolov8 for fiber bundle segmentation in x-ray computed tomography images of 2.5 d composites build the finite element model. Composites Part A: Applied Science and Manufacturing 185:108337, 2024b.
Yichi Zhang, Zhenrong Shen, and Rushi Jiao. Segment anything model for medical image segmentation: Current applications and future directions. Compulers in Biology and Medicine, pp. 108238, 2024c.
Zongjian Zhang, Yanli Zou, Yufei Tan, and Chiyang Zhou. Yolov8-seg-cp: a lightweight instance segmenta- tion algorithm for chip pad based on improved yolov8-seg model. Scientific Reports, 14(1):27716, 2024d.
Peng Zheng, Dehong Gao, Deng-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, and Nicu Sebe. Bilateral reference for high-resolution dichotomous image segmentation. CAAI Artificial Intelligence Research, 3: 9150038, 2024. https://hugging: http:///ugggggggecccoooo////////
Lei Zhu, Wenzhe Gu, Chengyong Liu, Beiyan Zhang, Wentao Liu, and Chaofeng Yuan. Image segmentation algorithm based on improved yolov8 model and its application in underground coal and gangue recognition. PLoS One, 2035)..0,, 2025.
25


Reviews:
[{"review_id": "SoSjGE2AL2", "reviewer": "Reviewer_f9h5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript presents a large-scale empirical study on prompt design for the Segment Anything Model (SAM), specifically applied to the task of segmentation refinement. The authors systematically evaluate 2,688 distinct prompt configurations, combining points, bounding boxes, and coarse masks with various augmentation strategies across four initial segmentation models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1.\tThe primary strength of this paper is its unprecedented scale and methodological rigor. Evaluating 10,752 unique model-prompt combinations provides a robust dataset from which to draw statistically significant conclusions, a clear advancement over prior ad-hoc studies.\n2.\tThe results are distilled into clear, practical guidelines. The key takeaways are well-supported by the data and presented with statistical significance tests (paired t-tests, confidence intervals), making the conclusions highly convincing.\n3.\tThe paper excels in its analysis of how different prompt components interact. This systemic approach is more insightful than studying components in isolation.\n4.\tThe study is well-designed, using multiple base models, datasets, and performance metrics (AP@50, 75, 95, 50-95) to demonstrate generalizability. The comparison against established methods like CascadePSP and SegRefiner provides crucial context for the claimed performance improvement.\n5.\tThe writing is generally clear. The prompt construction and augmentation methods are described in detail with mathematical formulations and algorithms, which enhances reproducibility.", "audience_yn": "Yes", "audience_explanation": "The findings are highly relevant to TMLR’s audience for several reasons. First, SAM is a widely used Foundational Model, so understanding how to effectively prompt it is of broad interest. Second, many real-world applications require high-precision segmentation, and model-agnostic refinement is a practical and active research area. Third, the scale and statistical rigor of the study address a clear gap in the literature and provide actionable insights for practitioners and researchers.", "requested_changes": "1.\tThe most significant limitation is the focus exclusively on scientific microscopy images (powder particles, cells). While this is a valuable and challenging domain, the title and abstract suggest broader applicability. The findings, particularly the strong negative effect of coarse masks, may not hold for natural images with different object characteristics (e.g., complex shapes, occlusions). It could be better to revise the title and introduction to more accurately reflect the domain-specific nature of the study.\n2.\tThe description of Distance Maximization and Voronoi placement algorithms (Algorithms 2 and 3) could be more intuitive. A brief high-level explanation before the pseudocode would improve readability.\n3.\tThe chosen ranges for parameters (e.g., 1-7 points, specific scaling factors for boxes/masks) are somewhat arbitrary. A brief justification for these ranges, perhaps based on pilot studies or heuristics, would strengthen the experimental design.", "broader_impact_concerns": "The paper raises no major ethical or societal concerns. The authors appropriately note the computational cost of SAM and recommend its use only when high precision is necessary. I think the work itself is methodological and domain-neutral, posing minimal risk of misuse.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "s3dBt9bNfk", "reviewer": "Reviewer_GYZY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Performs a large scale empirical study of the impact of prompting strategies on Segment Anything (SAM) segmentation refinement, across a wide range of prompting modalities, models, and datasets. The analysis reveals statistically significant impacts of including bounding boxes (positive) and coarse mask (negative) in the prompt.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper performs an incredibly comprehensive empirical study and rigorous statistical analysis and so backs up its claims very clearly. Its findings are concrete, prescriptive, and convincing, and likely to be of value to researchers and practitioners alike.", "audience_yn": "Yes", "audience_explanation": "SAM is used widely in the computer vision community, but has shown somewhat mixed results on scientific and microscopy datasets. By focusing its analysis on a microscopy dataset and further demonstrating generality to other microscopy datasets, this paper provides valuable insights towards extending the usefulness of SAM for scientific images.", "requested_changes": "This is a very well conducted and written empirical study. The only experiment I found lacking was demonstrating whether the proposed findings also extend to future generations of SAM (eg. SAM-2). It will be valuable to evaluate whether or not that holds true, and explore potential reasons.", "broader_impact_concerns": "No broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FvRD5aB3lp", "reviewer": "Reviewer_MMzu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an extensive empirical study on the Segment Anything Model （SAM）along with its prompt use. Specifically, 2,688 prompt configurations covering three different kinds of prompts are conducted, along with 10,752 evaluations. As a result, some valuable prompt configurations can be observed. The conclusion is valuable for future work to adapt SAM on the target domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The prompt configurations and evaluations in this paper are very extensive.", "audience_yn": "Yes", "audience_explanation": "The conclusion is valuable for future work to adapt SAM on the target domains.", "requested_changes": "- It is very clear that the proposed refinement framework contains many key steps or components, as shown in Fig.2. However, there is no explicit ablation study or table in this paper, that could systematically discuss and analyze the impact of each step or component.\n\n- Besides, this paper lacks visual prediction results under these different settings with different steps.\n\n- More segmentation results from various image domains, e.g., different medical image modalities, driving scene images, should be provided, to convince the readers with the huge working load and configurations. \n\n- Some recent surveys that analyze SAM on multiple domains are missing, for example:\n[1] Ji, Wei, et al. \"Segment anything is not always perfect: An investigation of sam on different real-world applications.\" (2024): 617-630.\nPlease enrich the related work and highlight the difference over the prior works.\n\n- For the limitation subsection Sec.5.4, it would be more convincing to also give some failure cases on each scenario.\n\n- For the results in Table 3, a more detailed discussion should be presented, to discuss why the proposed method degrades when compared with other existing solutions.\n\n- For the results and analysis in Sec.5.2, could the authors maybe provide some visual examples and illustrations for each subsection, so as to enhance the clarity?\n\n- I think the authors use a low-resolution version of Fig.5. Please use a high-resolution file to compile the submission.", "broader_impact_concerns": "N/A", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)