SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Ablation study of the HQ-Output Token on four extremely fine-grained segmentation datasets. We adopt the boxes converted from their GT masks as the box prompt input. By default, we train the predicted mask of HQ Output-Token by computing full GT mask loss.
Model	DIS [35]	COIFT [29]	HRSOD [51]	ThinObject [29]	Average
mIoU	mBIoU	mIoU	mBIoU	mIoU	mBIoU	mIoU	mBIoU	mIoU	mBIoU
SAM (baseline)	62.0	52.8	92.1	86.5	90.2	83.1	73.6	61.8	79.5	71.1
Using SAM's mask decoder feature:
SAM + Context Token [56]	71.5	62.2	93.0	87.7	91.8	85.0	84.5	73.1	85.2	77.0
SAM + HQ-Output Token (× Output Token)	75.1	65.8	93.9	88.9	93.0	86.1	86.1	74.6	87.0	78.9
SAM + HQ-Output Token (Boundary Loss)	75.2	66.4	94.0	88.9	92.1	85.7	87.3	76.0	87.2	79.3
SAM + HQ-Output Token	75.3	66.0	94.2	89.2	93.0	86.1	86.8	75.4	87.3	79.2
Using Our HQ-Feature:
SAM + HQ-Output Token (+ Context Token)	78.5	70.4	94.6	89.6	93.6	87.0	88.9	79.3	88.9	81.6
SAM + HQ-Output Token	78.6	70.4	94.8	90.1	93.6	86.9	89.5	79.9	89.1	81.8
Table 3: Ablation study on the HQ-Features sources. Early-layer denotes the feature after the first global attention block of the ViT encoder, while final-layer denotes the output of the last ViT block. Four HQ datasets denote DIS (val) [35], ThinObject-5K (test) [29], COIFT [29] and HR-SOD [51].
Model	Fusion conv	Decoder Mask feature	ViT Encoder	Four HQ datasets
Final-layer	Early-layer	mIoU	mBiOU
SAM [21]	✓	79.5	71.1
HQ-SAM (Ours)	✓	87.3	79.2
✓	✓	87.8	80.1
✓	✓	15.1	9.0
✓	✓	✓	88.6	81.3
✓	✓	✓	88.6	81.1
✓	✓	✓	✓	89.1	81.8
four HQ datasets but leads to very poor performance on COCO, indicating strong overfitting. We also observe a similar phenomenon when directly finetuning SAM's mask decoder. Only finetuning SAM's output token can address the catastrophic forgetting problem with improvement on the four HQ datasets and COCO. However, the incremental improvement is still much smaller compared to ours. HQ-SAM improves  $1.1\mathrm{AP}_B$  on COCO while output token finetuning only gives an increase of  $0.4\mathrm{AP}_B$ . This shows the advantage of HQ-SAM in data-efficient learning while preserving the zero-shot capability of SAM.
(a) Recall comparison on COIFT
(b) Recall comparison on HRSOD
Figure 4: Recall rate comparison between COIFT [29] and HRSOD [51] under the zero-shot protocol, using BIoU thresholds ranging from loose to strict. The performance gap between SAM and our HQ-SAM increases significantly when we vary from a loose BIoU threshold of 0.5 to a very strict threshold of 0.9, showing the advantage of HQ-SAM in predicting very accurate segmentation masks.
Accuracy analysis at different BIoU thresholds Figure 4 compares SAM and HQ-SAM from loose to strict BIoU thresholds. We plot the percentage of mask predictions that have a BIoU larger than the threshold indicated on the x-axis. The large performance gap with strict IoU thresholds on both COIFT [29] and HRSOD [51] clearly validates the advantage of HQ-SAM in predicting very accurate masks. However, even at the loose threshold of 0.5, HQ-SAM reduces the number of incorrect predictions by SAM by  $81\%$  for COIFT and  $69\%$  for HRSOD. This shows that HQ-SAM predictions are not only substantially more accurate but also more robust in challenging cases.
Table 4: Comparison with model finetuning or extra post-refinement [6]. For the COCO dataset, we use a SOTA detector FocalNet-DINO [53] trained on the COCO dataset as our box prompt generator.
Model	Four HQ datasets	APB	AP	COCO APL	APM	APS
mIoU	mBIoU
SAM (baseline)	79.5	71.1	33.3	48.5	63.9	53.1	34.1
Training the whole SAM	38.0	12.2	0.2	5.5	-	-	-
Add Context Token [56]	85.2	77.0	31.9	47.2	65.1	51.2	31.9
CascadePSP Post-refinement [6]	80.9	74.6	2.8	13.4	43.4	9.4	0.0
CRM Post-refinement [37]	81.4	75.4	15.9	28.7	-	-	-
Finetune SAM's decoder	87.6	79.5	9.0	19.5	45.2	15.8	4.7
Finetune SAM's output token	87.6	79.7	33.7	48.7	66.0	52.3	33.6
HQ-SAM (Ours)	89.1	81.8	34.4	49.5	66.2	53.8	33.9
Table 5: Zero-shot open-world instance segmentation results comparison on UVO [42]. We use FocalNet-DINO [53] trained on the COCO dataset as our box prompt generator. *strict denotes the boundary region with a tighter threshold.
Model	APstrictB	APstrictB75	APstrictB50	APB	APB75	APB50	AP
SAM	8.6	3.7	25.6	17.3	14.4	37.7	29.7
HQ-SAM	9.9	5.0	28.2	18.5	16.3	38.6	30.1
Table 6: Zero-shot segmentation result comparison on the test set of high-quality BIG [6] benchmark using various types of input prompts. We employ PSPNet [55] to generate the coarse mask prompt.
Model	GT Box Prompt	Mask Prompt
mIoU	mBiIoU	mIoU	mBiIoU
SAM	81.1	70.4	66.6	41.8
HQ-SAM	86.0	75.3	86.9	75.1
4.3 Zero-shot Comparison with SAM
We perform extensive zero-shot transfer comparisons between our HQ-SAM and SAM on 7 benchmarks, including SGinW [58], COCO [31], UVO [42], LVIS [14], HQ-YTVIS [20], BIG [6], COIFT [29] and HR-SOD [51], where HQ-SAM outperforms SAM without bells and whistles, demonstrating its efficacy and kept generalization ability even trained with a small-scale dataset.
Results on the SGinW Benchmark Equipped with the same Grounding-DINO [32] as box prompts, we also performed experiments by replacing SAM with HQ-SAM in Grounded-SAM, and obtained the first place in the Segmentation in the Wild (SGinW) competition<sup>1</sup> on the zero-shot track. Note that SGinW contains 25 zero-shot in-the-wild segmentation datasets for evaluation, and Grounded-HQ-SAM with 49.6 mean AP and outperforms Grounded-SAM obviously using the same detector.
Zero-Shot Open-world Segmentation To evaluate the zero-shot segmentation results in the openworld environment, in Table 5, we compare SAM and our HQ-SAM on the challenging UVO [42] benchmark with diverse and dense objects mask annotations. By taking the same pre-trained object detector [53] as box prompt input, our HQ-SAM improves for  $1.3\mathrm{AP}_B^{\mathrm{strict}}$  and  $2.6\mathrm{AP}_{B50}^{\mathrm{strict}}$  over SAM.
Zero-Shot Segmentation on High-resolution BIG Dataset In Table 6, we compare the zero-shot segmentation quality between SAM and HQ-SAM on the high-resolution BIG benchmark [6] with two types of prompts, including using GT object boxes or the provided coarse masks input. HQ-SAM consistently surpasses SAM, with obvious advantages using different types of prompts, and is much more robust to coarse masks prompts with partial boundary errors (provided by PSPNet [55]).
Zero-shot Instance Segmentation on COCO and LVIS In Table 7, we also evaluate HQ-SAM on the popular COCO and LVIS benchmarks respectively by feeding box prompts generated by the trained detectors of these two datasets. HQ-SAM consistently outperforms SAM by  $1.1\mathrm{AP}_B$  on COCO and  $0.7\mathrm{AP}_{B75}^{\mathrm{strict}}$  on LVIS, showing the improved mask quality and well-preserved zero-shot segmentation ability during the HQ-SAM training process.
<sup>1</sup> SGinW Benchmark Results: https://eval.ai/web/challenges/challenge-page/1931/leaderboard/4567
Table 7: Zero-shot instance segmentation results comparison on COCO [31] and LVISv1 [14]. For the COCO dataset, we use FocalNet-DINO [53] detector trained on COCO. For LVIS, we adopt ViTDet-H [28] trained on the LVIS dataset as our box prompt generator. For SAM, we use the ViT-L backbone and box prompt. We maintain the zero-shot segmentation capability of the original SAM while improving the mask quality on the boundary region.
Model	COCO	LVIS
APB	AP	APstrictB	APstrictB75	APB	APB75	AP
SAM	33.3	48.5	32.1	32.8	38.5	40.9	43.6
HQ-SAM	34.4	49.5	32.5	33.5	38.8	41.2	43.9
(a) Performance comparison on COIFT
(b) Performance comparison on DIS
Figure 5: Interactive segmentation results comparison using a varying number of input points on the COIFT [29] (zero-shot) and DIS [35] val set. HQ-SAM consistently outperforms SAM with various point numbers, and the relative improvement is more obvious with less prompt ambiguity.
Table 8: Zero-shot Video Instance Segmentation comparison on the test set of the very accurately labeled HQ-YTVIS [20] benchmark. We utilize pre-trained Swin-L-based Mask2Fromer [4] on YTVIS [47] as our box prompt input while reusing its object association prediction.
Model	APB	AP75	AP50	APM	APM75	AP50
SAM	30.2	19.1	72.9	60.7	68.1	90.5
HQ-SAM	34.0	24.3	79.5	63.6	70.5	91.1
Point-based Interactive Segmentation Comparison To investigate the segmentation performance of HQ-SAM with interactive point prompts, in Figure 5, we compare HQ-SAM to SAM with varying numbers of input points on COIFT [29] (zero-shot) and DIS [35] val set. HQ-SAM consistently outperforms SAM with different point prompts on both two datasets. We note that the relative performance increase is more significant when the prompt contains less object ambiguity with more input points information (increasing from 1 positive point to 10 positive points + 5 negative points).
Zero-shot High-quality Video Instance Segmentation Besides conducting image-based segmentation evaluation, we also perform video instance segmentation results comparison on the accurately annotated HQ-YTVIS benchmark [20]. We take the pre-trained Mask2Former [4] as our video box prompts and feed it into SAM and our HQ-SAM for mask prediction. In Table 8, HQ-SAM achieves remarkable gains of 3.8 points in Tube Boundary  $\mathrm{AP}^B$  and 2.9 Tube Mask  $\mathrm{AP}^M$ .
Visualization of HQ-Output Token In Figure 6, we provide visual comparison of our HQ-Output Token vs. SAM's common output token for their cross-attention maps in the last token-to-image layer of the mask decoder. We observe that our HQ-Output Token attends to the boundary and thin structure regions that are missed by the common token.
Zero-shot Visual Results Comparison In Figure 7, we compare HQ-SAM to SAM qualitatively in a zero-shot transfer setting, where HQ-SAM significantly promotes the mask details of SAM and also improves the masks of broken holes or large portion errors by the enriched semantic context. Refer to the supplemental file for more visual comparisons.
Comparison with Adapter Tuning Strategy In Table 9, we also compare our efficient token adaptation strategy to the recent Adapter Tuning [48] and LoRA [17]. We introduce lightweight adapters to ViT layers of SAM's encoder for encoder tuning and identify that this strategy leads to overfitting and its zero-shot performance on COCO decreases from 33.3 to 29.6. This validates our design choice to freeze SAM's encoder, and mainly focus on SAM's decoder.
Figure 6: Cross-attention of SAM's original token vs. HQ-Output Token in the last decoder layer. HQ-Token attends to the boundary and thin structure regions that are missed by the original token.
Figure 7: Visual results comparison between SAM (top row) vs. HQ-SAM (bottom row) in a zero-shot transfer setting, given the same red box or point prompt. HQ-SAM produces significantly more detailed-preserving results and also addresses the mask errors with broken holes.
Table 9: Comparison to Adapter Tuning [48] or using LoRA [17] in SAM's encoder using ViT-L based SAM and the same HQSeg-44K. For the COCO dataset, we use the SOTA detector FocalNet-DINO [53] trained on the COCO dataset as our box prompt generator.
Model	COCO	Model Params (MB)
APB	AP	APL	APM	APS	Total	Trainable
SAM	33.3	48.5	63.9	53.1	34.1	1191	-
SAM + LoRA [17]	28.6	43.7	-	-	-	1192.5	1.5
SAM + Encoder Adapter [48]	29.6	44.8	63.9	47.8	29.0	1203	12.0
HQ-SAM	34.4	49.5	66.2	53.8	33.9	1196.1	5.1
Mobile Efficiency Although HQ-SAM significantly boosts SAM's mask quality with negligible overhead, it shares the heavy ViT encoder of SAM, and thus cannot achieve a real-time speed in video processing. For efficient mobile deployment, we propose Light HQ-SAM based on the tiny ViT image encoder provided by MobileSAM [52]. In Figure 2, achieving running speed of 41.2 FPS, Light HQ-SAM improves the zero-shot COCO AP of MobileSAM from 44.3 to 45.0 with negligible additional cost, i.e., 1.7MB increase in model parameters.
We propose HQ-SAM, the first high-quality zero-shot segmentation model by introducing negligible overhead to the original SAM. We propose a lightweight High-quality Output Token in HQ-SAM to replace the original SAM's output token for high-quality mask prediction. After training only on 44K highly-accurate masks, HQ-SAM significantly boosts the mask prediction quality of SAM, which was trained on 1.1 billion masks. The zero-shot transfer evaluation is performed on 8 segmentation benchmarks across both image and video tasks, spanning diverse objects and scenes. Our research offers timely insights into how to leverage and extend SAM-like foundational segmentation models in a data-efficient and computation-affordable manner.
[56] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 2022.
[57] Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot semantic segmentation. In CVPR, 2023.
[58] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. In CVPR, 2023.
[59] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. In NeurIPS, 2023.
Supplementary Material: Segment Anything in High Quality
In this supplementary material, Section 6 first presents the additional experimental analysis of our HQ-SAM, including more zero-shot transfer comparisons to SAM on both image and video benchmarks. Then, in Section 7, we describe more details of our method implementation, including the training and inference. In Section 8, we provide further details of our constructed HQSeg-44K dataset for training HQ-SAM. In Section 9, we show extensive visual results comparison between our HQ-SAM and SAM on COCO [31], DIS-test [35], HR-SOD [51], NDD20 [41], DAVIS [34], and YTVIS [47].
6 Supplementary experiments
SAM vs. HQ-SAM on Various Backbones In Table 10, we provide a comprehensive comparison between HQ-SAM and SAM using various backbones, including ViT-B, ViT-L, ViT-H and TinyViT. The comparison not only includes the numerical results on the four HQ datasets and COCO validation set, but also contains the model sizes/speed/memory. HQ-SAM consistently outperforms SAM using three different backbones, with over 10 points increase in mBIoU on the four HQ datasets. Notably, the ViT-B based HQ-SAM significantly improves the  $\mathrm{AP}^B$  on COCO from 28.2 to 31.3 and AP from 44.4 to 46.7, with only a  $1.1\%$  increase in model parameters and negligible extra memory consumption.
Table 10: SAM vs. HQ-SAM on various ViT backbones. For the COCO dataset, we use a SOTA detector FocalNet-DINO [53] trained on the COCO dataset as our box prompt generator.
Model	Four HQ datasets	APB	AP	COCO	APS	Model Params (MB)	FPS	Memory
mIoU	mBIoU	APL	APM	APS	Total	Learnable
SAM-B	70.6	62.3	28.2	44.4	57.7	48.7	32.1	358	358	10.1	5.1G
HQ-SAM-B	86.3	78.1	31.3	46.7	62.9	50.5	32.0	362.1	4.1	9.8	5.1G
SAM-L	79.5	71.1	33.3	48.5	63.9	53.1	34.1	1191	1191	5.0	7.6G
HQ-SAM-L	89.1	81.8	34.4	49.5	66.2	53.8	33.9	1196.1	5.1	4.8	7.6G
SAM-H	75.6	68.3	34.0	48.9	64.5	53.3	34.4	2446	2446	3.5	10.3G
HQ-SAM-H	89.3	81.5	34.9	49.9	66.5	54.0	34.2	2452.1	6.1	3.4	10.3G
MobileSAM	69.0	58.8	28.6	44.3	-	-	-	38.6	38.6	44.8	3.7G
Light HQ-SAM	81.4	71.6	29.6	45.0	-	-	-	40.3	1.7	41.2	3.7G
Table 11: Results on YouTubeVIS 2019 validation set and HQ-YTVIS test set using ViT-L based SAM. We adopt the SOTA detector Mask2Former [4] trained on the YouTubeVIS 2019 dataset as our video boxes prompt generator while reusing its object association prediction.
Model	YTVIS 2019	HQ-YTVIS
AP	\(AP_{50}\)	\(AP_{75}\)	\(AP_L\)	\(AP_M\)	\(AP_S\)	\(AP^B\)	\(AP^M\)
SAM	51.8	82.1	55.4	65.5	52.0	34.2	30.2	60.7
HQ-SAM	53.2	82.9	58.3	66.4	53.3	33.7	34.0	63.6
Zero-shot Video Instance Segmentation Comparison Extending from Table 8 of the paper (evaluation on the HQ-YTVIS benchmark [20]), we further perform a comparative analysis of zero-shot video instance segmentation results on the popular YTVIS 2019 [47] validation set. We take the pre-trained Mask2Former [4] as our video box prompts and feed them into SAM and our HQ-SAM for mask prediction. In Table 11, HQ-SAM achieves consistent gains of 1.4 points in Tube Mask AP, increasing SAM's performance from 51.8 to 53.2. Interestingly, we find the  $\mathrm{AP}_{75}$  improvement with a higher IoU threshold for HQ-SAM is much larger than  $\mathrm{AP}_{50}$ , further validating the advantages of HQ-SAM in high-quality mask prediction.
Zero-shot Video Object Segmentation Comparison Besides video instance segmentation, in Table 12, we further report the comparison of video object segmentation results between HQ-SAM and SAM on DAVIS validation set in a zero-shot transfer protocol. We take the pre-trained XMem as our video box prompts and feed the same prompts into SAM and HQ-SAM. HQ-SAM improves SAM the  $\mathcal{I}\& \mathcal{F}$  from 82.0 to 83.2 and the  $\mathcal{F}$  score from 84.9 to 86.1, where  $\mathcal{F}$  is for measuring the contour accuracy of the video objects.
Table 12: Results on DAVIS 2017 [34] validation set using ViT-L based SAM. We adopt the SOTA model XMem [7] as our video boxes prompt generator while reusing its object association prediction.
Model	J&F	J	F
SAM	82.0	79.0	84.9
HQ-SAM	83.2	80.3	86.1
Robustness to Input Box Prompts In Table 13, we compare HQ-SAM to SAM by adding various scales of noises to the input ground truth box prompts. In practice, we cannot expect the input box prompts provided by humans in interactive modes to be identical to the ground truth (GT) boxes or extremely accurate. We follow the data augmentation code in DN-DETR [25] to add different noise scales and identify that our HQ-SAM is much more robust compared to SAM, where the relative mBIoU advantage improves from 10.7 to 20.5 when gradually increasing the noise scales. Note that our method is not trained with noised boxes. We also visualize such noised input case in Figure 11, where SAM is more sensitive to small box location shifts that easily happened during interactive annotation.
Table 13: Comparison of segmentation accuracy on the four HQ datasets by adding various noise levels to the GT box prompts input.
Model	No Noise	Noise scale 0.2	Noise scale 0.4
mIoU	mBIoU	mIoU	mBIoU	mIoU	mBIoU
SAM	79.5	71.1	65.7	57.1	46.4	39.8
HQ-SAM	89.1	81.8↑10.7	82.8	73.4↑16.3	69.9	60.3↑20.5
7 Additional Implementation details
Training Details During training HQ-SAM on the composed HQSeg-44K, we fix the model parameters of the pre-trained SAM model while only making the proposed HQ-SAM learnable, including HQ-Output Token, its associated three-layer MLP and three convolutions for HQ-Features fusion. Two of them are transposed convolutions (size  $2 \times 2$ , stride 2) used to upscale encoder embedding size from  $64 \times 64$  to  $256 \times 256$ . We treat the new HQ-Output Token as the fifth mask token compared to the original four mask tokens in SAM's mask decoder. During training, this new HQ-Output token of size  $1 \times 256$  is concatenated with SAM's mask tokens (size of  $4 \times 256$ ), iou token (size of  $1 \times 256$ ) and prompt tokens (size of  $\mathrm{N}_{\mathrm{prompt}} \times 256$ ) as the input to the SAM's mask decoder. For example, if the input image contains  $N$  box prompts (size  $\mathrm{N} \times 2 \times 256$ ), the final concatenated input and output shape for the 2-layer mask decoder of SAM is  $\mathrm{N} \times (1 + 4 + 1 + 2) \times 256$ . For experiments using ViT-B, ViT-L, and ViT-H-based models on training, we adopt the same training setting, with a learning rate of 1e-3 and train our HQ-SAM for 12 epochs (learning rate drops to 1e-4 after 10 epochs). We supervise mask prediction of the new HQ-Output token with a combination of both BCE Loss and Dice Loss.
Implementation Details We follow the same inference pipeline of SAM but use the mask prediction from HQ-Output token as high-quality mask prediction. Table 10 reports the detailed inference speed comparison using various backbones. For box-prompting-based evaluation, we feed SAM and our HQ-SAM with the same image/video bounding boxes and adopt the single mask output mode of SAM. For interactive segmentation comparison using a single point, we follow SAM and adopt the "center" point of Ground Truth (GT) masks, which is at a maximal value location in a mask's interior distance transform. For multiple-point evaluation, we randomly sample the points from the GT masks and report the averaged results with three trials.
8 More Details of HQSeg-44K
Data composition of HQSeg-44K In Table 14, we provide more details of our composed new training dataset HQSeg-44K which contains 44,320 extremely accurate image mask annotations, where we show their annotation quality in Figure 8. HQSeg-44K is a collection of six existing image datasets including DIS [35] (train set), ThinObject-5K [29] (train set), FSS [26], ECSSD [38], MSRA-10K [8], DUT-OMRON [46] with extremely fine-grained mask labeling, where each of them contains 7.4K mask labels on average. This composed training set has no images/annotations overlapping with the zero-shot evaluation datasets adopted in our paper.
Effect of HQSeg-44K In Table 15, we show the advantage of using HQSeg-44K by comparing HQ-SAM training with 44K randomly sampled images and masks from SA-1B [21]. Using the same efficient token learning strategy, training with SA-1B (44K) decreases the averaged mBiOU on the four datasets from 71.1 to 70.1, while ours improves it from 71.1 to 81.8. This validates the effectiveness of our constructed HQSeg-44K benchmark in improving mask quality. Note that the ablation experiments in Table 2, Table 3, Table 4, and Table 9 of the paper are all based on the constructed HQSeg-44K.
Table 14: Data composition of our constructed HQ-Seg-44K.
Dataset	DIS [35]	Thin-Object 5k [29]	FSS [26]	DUTS [46]	ECSSD [38]	MSRA-10K [8]	Total
Image Num.	3000	4748	10000	15572	1000	10000	44320
DUT-OMRON MSRA-10K ECSSD FSS-1000 ThinObject-5K DIS
Figure 8: Visualization of annotated mask quality for randomly selected cases from the six dataset components of the HQ-Seg-44K. Zoom in for better viewing the fine-grained mask details.
Zero-shot results on DIS and ThinObject-5K We also report zero-shot results in Table 16 on DIS and ThinObject-5K by removing the training splits of either or both datasets from the training of
Table 15: Comparison of the training dataset. For the COCO dataset using ViT-L-based SAM, we use a SOTA detector FocalNet-DINO [53] trained on the COCO dataset as our box prompt generator.
Model	Dataset	DIS	COIFT	HRSOD	ThinObject	Average
mIoU	mBIoU	mIoU	mBIoU	mIoU	mBIoU	mIoU	mBIoU	mIoU	mBIoU
SAM	SA-1B	62.0	52.8	92.1	86.5	90.2	83.1	73.6	61.8	79.5	71.1
HQ-SAM	+ SA-1B-44K	60.4	51.7	91.1	86.1	88.4	80.9	73.1	61.8	78.3	70.1
HQ-SAM	+ HQ-Seg-44K (Ours)	78.6	70.4	94.8	90.1	93.6	86.9	89.5	79.9	89.1	81.8
Figure 9: Visual results comparison between SAM (top row) vs. HQ-SAM (bottom row) on DIS test set, given the same red box prompt. HQ-SAM produces significantly more accurate boundaries.
HQ-SAM. The improvement of HQ-SAM over SAM is still substantial on DIS or ThinObject (over 10.0 points on DIS-mIoU and 9.0 points on ThinObject-mIoU), even when the corresponding training splits are removed from training.
Table 16: Zero-shot results on DIS and ThinObject-5K by removing the training splits of either or both datasets from the training of HQ-SAM. Results not obtained in a zero-shot manner (i.e. the training split was used), are shown in parenthesis to easily compare zero-shot results.
Training Setting	DIS-mIoU	DIS-mBIoU	ThinObject-mIoU	ThinObject-mBIoU
SAM (baseline)	62.0	52.8	73.6	61.8
HQ-SAM (remove both DIS and ThinObject)	72.9	63.1	82.7	70.7
HQ-SAM (remove DIS)	74.7	66.2	(90.1)	(80.4)
HQ-SAM (remove ThinObject)	(78.4)	(70.3)	83.3	72.1
HQ-SAM (default HQSeg-44K)	(78.6)	(70.4)	(89.5)	(79.9)
9 More Visual Results Comparison