Segment Anything in High Quality
Authors: Lei Ke
Mingqiao Ye
Martin Danelljan
Yifan liu
Yu-Wing Tai
Chi-Keung Tang
Fisher Yu
Venue: NeurIPS 2023 poster (2023)
Keywords: segment anything
zero-shot segmentation
high-quality segmentation
TL;DR: We propose HQ-SAM to upgrade SAM for high-quality zero-shot segmentation.

Abstract: The recent Segment Anything Model (SAM) represents a big leap in scaling up segmentation models, allowing for powerful zero-shot capabilities and flexible prompting. Despite being trained with 1.1 billion masks, SAM's mask prediction quality falls short in many cases, particularly when dealing with objects that have intricate structures.  We propose HQ-SAM, equipping SAM with the ability to accurately segment any object, while maintaining SAM's original promptable design, efficiency, and zero-shot generalizability. Our careful design reuses and preserves the pre-trained model weights of SAM, while only introducing minimal additional parameters and computation. We design a learnable High-Quality Output Token, which is injected into SAM's mask decoder and is responsible for predicting the high-quality mask. Instead of only applying it on mask-decoder features, we first fuse them with early and final ViT features for improved mask details. To train our introduced learnable parameters, we compose a dataset of 44K fine-grained masks from several sources. HQ-SAM is only trained on the introduced detaset of 44k masks, which takes only 4 hours on 8 GPUs. We show the efficacy of HQ-SAM in a suite of 10 diverse segmentation datasets across different downstream tasks, where 8 out of them are evaluated in a zero-shot transfer protocol. Our code and pretrained models are at https://github.com/SysCV/SAM-HQ.

Segment Anything in High Quality
Lei Ket
Mingqiao Ye"
Martin Danelljan' !
Yifan Liu'
Yu-Wing Tai
Chi-Keung Tang²
Fisher Yu'
ETH Zürrch
²HKUST
DDatmouth College
Abstract
The recent Segment Anything Model (SAM) represents a big leap in scaling up segmentation models, allowing for powerful zero-shot capabilities and flexible prompting. Despite being trained with 1.1 billion masks, SAM's mask prediction quality falls short in many cases, particularly when dealing with objects that have intricate structures. We propose HQ-SAM, equipping SAM with the ability to accurately segment any object, while maintaining SAM's original promptable design, efficiency, and zero-shot generalizability. Our careful design reuses and preserves the pre-trained model weights of SAM, while only introducing minimal additional parameters and computation. We design a learnable High-Quality Output Token, which is injected into SAM's mask decoder and is responsible for predicting the qush--uultty mask. Instead of only applying it on mecsdee features, we first fuse them with early and final ViT features for improved mask details. To train our introduced learnable parameters, we compose a dataset of 44K fine-grained masks from several sources. HQ-SAM only trained on the introduced detaset of 44k masks, which takes only 4 hours on 8 GPUs. We show the efficacy of HQ-SAM in a suite of 10 diverse segmentation datasets across different downstream tasks, where 8 out of them are evaluated in a zero-shot transfer protocol. Our code and pretrained models are at https: pettpa://gitie //github. /atp//yy/////aa HS.
Introduction
Accurate segmentation of diverse objects is fundamental for a wide range of scene understanding applications, including image/video editing, robotic perception, and AR/VR. Trained with billion- scale mask labels, the Segment Anything Model (SAM) [21] was recently released as foundational vision model for general image segmentation. SAM is capable segmenting wide range of objects, parts, and visual structures in diverse scenarios, by taking prompt consisting of points, a bounding box, or a coarse mask as input. Its zero-shot segmentation abilities have led to a rapid paradigm shift, as it can be transferred to numerous applications through simple prompting.
While SAM has achieved impressive performance, segmentation results are still unsatisfactory in many cases. In particular, SAM suffers from two key problems: 1) Coarse mask boundaries, often even neglecting the segmentation of thin object structures, as shown in Figure 1. 2) Incorrect predictions, broken masks, large errors in challenging cases. This is often related to SAM misinterpreting thin structures, such as the kite lines in the rightmost column of Figure 1. These types of failures severely limit the applicability and effectiveness of foundational segmentation models, such as SAM, in particular for automated annotation and image/video editing tasks, where highly accurate image masks crucial.
We propose HQ-SAM, which can predict highly accurate segmentation masks, even in very challeng- ing cases (see Figure 1), without compromising the strong zero-shot capabilities and flexibility of the
'Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

SAM Prediction
HQ-SAM Prediction
Figure 1: The predicted masks of SAM vs. HQ-SAM, given the same red box several points on the object as input prompts. HQ-SAM produces significantly more detailed results with very accurate boundaries. In the rightmost column, SAM misinterprets the thin structure of the kite lines, and produces a large portion of errors with broken holes for the input box prompt.
original SAM. To preserve the efficiency and zero-shot performance, we propose minimal adaptation of SAM, adding less than 0.5% parameters, to extend its capability to high-qualit segmentation.
Directly fine-tuning the SAM decoder introducing new decoder module severely degrades the general zero-shot segmentation performance. We therefore propose the HQ-SAM architecture, which tightly integrates with and re-uses the existing learned SAM structure, in order to fully preserve the zero-shot performance. First, we design a learnable HQ-Output Token that is input to SAM's mask decoder, alongside the original prompt and output tokens. Unlike the original output tokens, our HQ-Output Token and its associated MLP layers are trained to predict a high-quality segmentation mask. Second, instead of only re-using the SAM's mask decoder features, our HQ-Output Token operates on a refined feature set to achieve accurate mask details. In particular, we use both global semantic context and local fine-grained features by fusing SAM's mask decoder features with early and late feature maps from its ViT encoder. During training, we freeze the entire pre-trained SAM parameters, while only updating our HQ-Output Token, its associated three-layer MLPs, and a small feature fusion block.
Learning accurate segmentation requires dataset with accurate mask annotations of diverse objects with complex and detailed geometries. SAM is trained on the SA-IB dataset, which contains IM images with 1.1 billion masks automatically generated by a SAM-like model. However, us- ing this extensive dataset presents significant cost implications and falls short of achieving the de- sired high-quality mask generations pursued in our work, evident by SAM's performance in Fig- ure 1. Consequently, we compose new dataset, called HQSeg-44K, which contains 44K extremely fine-grained image mask annotations. HQSeg- 44K is constructed by merging six existing image datasets [35, 29, 26, 38, 8, 46] with highly accurate mask labels, covering over 1,000 diverse semantic classes. Thanks to the smaller-scal scale dataset and our minimal integrated architecture, HQ-SAM can be trained in only 4 hours on 8 RTX 3090 GPUs.
HO-SAM (VITH
50
MITLY
47.54
(VIT-B)
Zero-shot COCO
Light HO-SAM (TinMMi)
SAM
(MT-8)
45-
MobileSAM (TinyMI)
42.5
40
500
1500
3000
Model Size [M]
Figure 2: Performance speed vs. model size for an array SAM variants 52].
To validate the effectiveness of HQ-SAM, we perform extensive quantitative and qualitative experi- wewttormanncsspeedd-eeeeee
size comparison on SAM variants [21, 52] in Figure 2. We compare HQ-SAM with SAM on suite of 10 diverse segmentation datasets across different downstream tasks, where 8 out them are under zero-shot transfer protocol, including COCO [31], UVO [42], SGinW [58], LVIS [14], HQ- NT-YIVIS [20], BIG [6], COIFT [29]
mental analysis. We provide a comprehensive

---

and HR-SOD [51]. This rigorous evaluation demonstrates that the proposed HQ-SAM can produce higher-quality masks while maintaining the zero-shot capability compared with SAM.
2 Related Work
High-quality Segmentation Existing works for high-quality segmentation are mostly trained for a specific segmentation task, like image and video instance segmentation [22, 19, 20, 40, 44], semantic segmentation [30, 54, 39, 50] or panoptic segmentation [9], in a close-world paradigm. Some of them focus on post-segmentation refinement using with graphical models such as CRF [23] region growing [10]. However, the CRF-based refinement is adhere to low-level color boundaries without fully utilizing high-level semantic context and cannot fix large segmentation errors. While some refinement-base based works adopt separate deep networks for cascade iterative refinement [6, 37], they are prone to overfitting as shown by our experiment. Compared these high-quality segmentation [19, 22, 33] or segmentation refinement methods, we focus on accurately segmenting diverse objects on new data with flexible prompting, and build high-quality zero-shot segmentation model that generalizes to various segmentation tasks and domains. Unlike the post segmentation refinmment works [6, 37]. to preserve the zero-shot segmentation capability of SAM. HQ-SAM predicts the new high-quality mask directly by reusing the image encoder and mask decoder of SAM, instead of taking the coarse mask and images as the input and feeding it into a separate refinement network. The model architecture of HQ-SAM builds upon SAM with negligible overhead, where we propose efficient token learning for accurate mask predictions. This is completely different from previous high-quality segmentation works, and we show its effectiveness across a wide range of zero-shot experiments.
Fine-tuning and Prompt Tuning for Foundation Models Foundation models [2, 1] first appear in the NLP community, where large language models such as GPT series [2] show strong zero-shot generalization to unseen tasks and data. Then, some prompt-based learning works [16, 27, 17] are proposed to help these pre-trained models generalize to the downstream tasks instead of fine-tuning the internal model parameters [15] for better transfer learning. For vision- based foundation models [21, 43, 591, prompt engineering [56, 45, 49, 57] that freezes the pre-trained model is first explored in vision-language models, such as CLIP [36]. These prompts with learnable parameters are designed to help downstream tasks with better context optimization. Different from the existing prompt-based or finetuning works, we focus on the minimal adaptation of SAM toward high-quality segmentation. We directly use the proposed HQ-Output Token output for accurate mask prediction, instead of only leveraging some learnable parameters [56] to help context learning and better generalization.
3 Method
We propose HQ-SAM to upgrade SAM for high-quality zero-shot segmentation. HQ-SAM is lightweight and only introduces two important adaptations to the SAM model. In Sec 3.1, we first briefly review the architecture of SAM on which HQ-SAM is built. Then, in Sec 3.2, we introduce our HQ-SAM with High- Quality Token (HQ-Output Token) and Global-local Feature Fusion, which are the key components to achieve better segmentation quality for SAM while preserving its zero-shot capability. Finally, in See 3.3, we describe the training and inference process of HQ-SAM, which is both data and computationally efficient.
3.1 Preliminaries: SAM
SAM [21] is composed of three modules: (a) Image encoder: a heavy ViT-based backbone for image feature extraction, resulting in image embedding in spatial size 64x64. (b) Prompt encoder: encoding the interactive positional information from the input points/boxes/maske to provide for the mask decoder. (c) Mask decoder: a two-layer transformer-based decoder takes both the extracted image embedding with the concatenated output and prompt tokens for final mask prediction. The released SAM model is trained on the large-scale SA- IB dataset, which contains over 1 billion automatically generated masks (400x more masks than any existing segmentation datasets [14, 24]) and 11 million images. Thus, SAM shows valuable strong zero-shot generalization to new data without the necessity for additional training. However, we also note that SAM training is very ex expensive, where distributively training ViT-H-based SAM for epochs on SA-1B requires 256 GPUs with a large batch size of 256 images. For more SAM method details, we refer readers to [21].
3

---

SAM
S Mask
Image Encoder
64x64
Decoddr Laver
Transposed
Mask foot.
Conv.
Prompt Token
Token to image atmn.
Oulput Token
(256)
Mask Decoder
MLP
Output Token
HQ-SAM Mask
HQ-SAM
Updated
HQ-Q Oipuut
Foot.
HQ-Output
Token
Token
Input Image
Error
Fest.
MLP
ICorrection
Early Layer
Globsi-los -ocal
HQ-Features
Point-wise Product
Final Laver
Fusion
256x226
Figure HQ-SAM introduces HQ-Output Token and Global-local Feature Fusion to SAM for high-quality mask prediction. To keep the sero-shot capability of SAM, the lightweight HQ-Output Token reuses SAM's mask decoder, and generates new MLP layers for performing point-wise product with fused HQ-Features. During training, only few learnable parameters in HQ-SAM are trainable while we fix the model parameters of the pre-trained SAM. The prompt encoder is omitted here for clarity. Error correction is simply used as direct element-wis sum between the predicted logits of the SAM's Output Token and the HQ-Output Token during inference.
3.2 Ours: HQ-SAM
In this section, we describe the architecture of the HQ-SAM network. To preserve the zero-shot transfer capability of SAM, while preventing model overfitting catastrophic forgetting, instead of directly finetuning SAM or adding new heavy decoder network, we take a minimal adaptation approach much as possible. To this end, HQ-SAM reuses the pre-trained model weights of SAM as much as possible with only two new key components, namely, High-Quality Output Token and Global-local Feature Fusion, illustrated in Figure 3. HQ-SAM can thus be regarded high- quality zero-shot segmentation model evolved from SAM with negligible extra model parameters and computation cost.
3.2.1 High-Quality Output Token
We propose efficient token learning for improving the mask quality SAM. As shown in Figure 3, in SAM's original mask decoder design, the output token (similar to object query in DETR [3]) is adopted for mask prediction, which predicts dynamic MLP weights and then performs point-wise product with the mask features. To promote SAM's mask quality in HQ SAM, instead of directly taking SAM's coarse masks as input, we introduce the HQ-Output token and a new mask prediction layer for high-quality mask prediction.
In Figure 3, by reusing and fixing SAM's mask decoder, new learnable HQ-Output Token (size of I x 256) is concatenated with SAM's output tokens (size of 4x26) and prompt tokens (size of Nprompt x 256) as the input the SAM's s mask decoder. Similar the original output token, in each attention layer, HQ-Output Token first performs self-attention with other tokens and then conducts both loken-to-immae and the reverse image-to-token attention for feature updating. Note that HQ-Output Token uses the point-wise MLP shared by the other tokens in each decoder layer. After passing through two decoder layers, the updated HQ-Output Token has access to the global image context, the critical geometric/type information of prompt tokens well hidden mask information of the other output tokens. Finally, we add a new three-layer MLP to generate dynamic convolutional kernels from the updated IiQ-Output Token, which then performs spatially point-wise product with the fused HQ-feature for high-quality mask generation.
Instead of directly finetuning SAM or further adding heavy post-refinement network, we only allow the HQ-Output Token and its associated three-layer MLPs to trained for correcting the mask errors of SAM's output token. This is completely different from existing high-quality segmentation models [19, 6, 20, 22]. We identify two main advantages of our efficient token learning through extensive experiments: 1) This strategy significantly improves SAM's 's mask quality while only

---

introducing negligible parameters compared to original SAM, making HQ-SAM training extremely time and data-efficient; 2) The learned token and MLP layers do not overfit to mask the annotation bias of specific dataset, thus keeping SAM's strong zero-shot segmentation capability on new images without catastrophic knowledge forgetting.
3.2.2 Global-local Fusion for High-quality Features
Very accurate segmentation also requires input image feature with both rich global semantic context and local boundary details. To further promote mask quality, we enrich both the high-level object context and low-level boundary/edge information in the mask decoder features of SAM. Instead of directly using SAM's mask decoder feature, we compose the new high-quality features (HQ- Features) by extracting and fusing features from different stages of the SAM model: 1) The early layer local feature of SAM's VIT encoder with spatial shape 64 64, which captures more general image edge/boundary details [12]. Concretely, we extract the feature after the first global attention block of the ViT encoder, and for ViT-Large based SAM, this is the 6th block output for the 24 blocks in total: 2) The final layer global feature of SAM's VIT encoder with shape 64 x 64, which has more global image context information; 3) The mask feature in SAM's mask decoder with size 256 x 226,, which is also shared by the output tokens, contains strong mask shape information.
As shown in Figure 3, to obtain the input HQ- Fe-auutures, we first upsample the early-layer and final- layer encoder features to the spatial size 256x256 by transposed convolution. Then, we sum up these three types of features in an element-wise manner after simple convolutional processing. We show that this global-local feature fusion is simple while effective, yielding detail preserving segmentation results with small memory footprint and computation burden. We also perform detailed ablation on the effect of each feature source in the experimental section (Table 3).
3.3 Training and Inference of HQ-SAM
Training Data Construction To train HQ-SAM in data-efficient manner, instead of further training on SA-IB [21], we compose a new training dataset HQSeg-44K which contains 44,320 extremely accurate image mask annotations. We note that the released SA-1B dataset only contains automatically generated mask labels, missing very accurate manual annotation on objects with complex structures. Due to the annotation difficulty. HQSeg-44K leverages collection of six existing image datasets including DIS [35] (train set), ThinObject-5K [29] (train set), FSS- 1000 [26], ECSSD [38], MSRA- 10K [8], DUT-OMRON [46] with extremely fineggained mask labeling, where each of them contains 7.4K mask labels on average. To make HQ-SAM robust and generalizable to new data, HQSeg-44K contains diverse semantic classes of more than 1, 1,000. We show the advantage of using HQSeg-44K by comparing HQ-SAM training with 44K randomly sampled images and masks from SA-1B [21] in our supplemental analysis.
HQ-SAM Training During training, we fix the model parameters of the pre-trained SAM model while only making the proposed HQ- HQ-SAM learnable. The learnable parameters thus only include the HQ-Output Token, its associated three-layer MLP and three simple convolutions for HQ- I[Q-Features fusion. Since SAM is designed for flexible segmentation prompts, we train HQ-SAM by sampling mixed types of prompts including bounding boxes, randomly sampled points, and coarse masks input. We generate these degraded masks by adding random Gaussian noise in the boundary regions of the GT masks. For generalizability to different object scales, we use large-scale jittering [13]. We use a learning rate of 0.001 and train We train on 8 Nvidia GeForce RTX 3090 GPUs with a total batch size of 32, which takes 4 hours to train for 16.6K iterations. Please refer our supplemental file for more details.
HAM for 12 epochs, with learning rate drop after 10 epochs.
HQ-SAM Inference We follow the same inference pipeline of SAM but use the mask prediction from HQ-Output token as high-quality mask prediction. During inference, we sum the predicted logits of the SAM mask (by Output Token) and our predicted mask (by HQ-Output Token) for mask correction spatial resolution 256x256. Then we up-sample the corrected mask to the original resolution 1024x 1024 as our output.
SAM vs. HQ-SAM on Training and Inference In Table 1, we report detailed training and inference comparisons between our HQ-SAM and SAM. While HQ-SAM produces substantially better segmentation quality, its training is very quick and affordable, which only takes 4 hours with RTX3090 GPUs. HQ-SAM is also lightweight and efficient, introducing negligible increases in model parameters, GPU memory usage, and inference time per image.
5

---

Table I: Training and inference comparison between ViT-L [11] based SAM and HQ-SAM. HQ-SAM brings negligible extra computation burden to SAM, with less than 0.5% increase in model parameters and reaching 96% of its original speed. SAM-L is trained on 128 A A100 GPUs for 180k iterations. Based on SAM-I, we only need to train our HQ-SAM on 8 RTX3000 GPUs for 4 hours.
Training # GPU
Inference
Method
Learnable Params (M)
Batch Size
Time (h)
FPS
Mem.
SAM [21]
1191
128
128
N/A
5.0
7.6G
HQ-SAM
5.1
8
32
4
4,8
7.6G
4 Experiments
4.1 Experimental Setup
Datasets For training we use the compiled HQSeg-44K, described in Section 3.3. For a comprehen- sive evaluation of the segmentation performance of HQ-SAM, we perform experiments on wide range of datasets, including four extremely fine-grained segmentation datasets: DIS [35] (validation set), ThinObject-5K [29] (test set), COIFT [29] and HR-SOD [51]. Besides, we experiment on popu- segmentation tasks in zero-shot settings, such as COCO [31], SGinW [58], UVO [42], LVIS [14), HQ-YTVIS [20] and BIG [6].
lar and challenging benchmarks across various image/video-
Evaluation Metries To accurately quantify improvements in mask quality, instead of only employing the standard mask AP or mask mloU, also adopt boundary metries mBloU and boundary APg [5]. We also evaluate on stricter AP by adjusting the default dilation ratio from 0.02 to 0.01 on UVO [42] and LVIS [14]. For evaluation on the four fine-graines segmentation datasets [35, 29, 51], we also report the averaged boundary and mask IoU among them. For video instance segmentation evaluation on HQ-YTVIL [20], we use both Tube Boundary AP and Tube Mask APM
4.2 Ablation Experiments
We conduct detailed ablation studies on the proposed HQ-SAM using ViT-Large as the backbone, analyzing the impact of the proposed HQ-Output Token and HQ- segmentation quality especially in zero-shot cases. For ablation experiments, we use the four aforementioned extremely accurate segmentation datasets, namely, DIS (val) [35], ThinObject-5K (test) [29], COIFT [29] and HR-SOD [51] as well as the COCO validation set.
Effect of the High-Qualit Output Token HQ-SAM employs HQ-Output Token for mask prediction. Table compares our HQ-Output Token the baseline SAM and other existing prompt/token learning strategies, such adding additional three context tokens [56] as learnable vectors into the SAM's mask decoder for better context learning. Compared to using context tokens, the HQ-Output token consistently brings larger performance gains four high-quality datasets, with 13.2 mBloU on DIS and 2.7 mBloU on COIFT datasets. We also perform other ablation experiment variants, such computing the scaled dot product [18] between the original SAM's output token and our HQ-Output token or restricting the mask loss to only inside the boundary regions, and find they slightly decrease the averaged performance HQ-SAM significantly improves the mBloU on DIS benchmark from 52.8 to 70.4 and also promotes the mBloU on the HRSOD dataset for 3.8 points.
the four evaluation datasets. Compared to SAM,
Ablation on the Global-local Fusion for HQ-Features Table 3 tabulates the effect of global-local fusion, where the importance of each feature component is analyzed in HQ-Features during the fusion process. Compared to directly using the mask decoder feature of SAM, the entire HQ-Features bring an obvious advantage of 2.6 mBloU four highly accurate segmentation datasets. The final-layer ViT encoder feature with global context increases the mBloU from 80.1 to 81.3. while the carly-layer feature with local details further promotes the mBloU to 81.8. We also replace the proposed global-local fusion with the conventional FPN to build a feature pyramid for fusion, and found this brought inferior performance, decreasing from 89.1 87.4 mloU.
Comparison to SAM finetuning or post-refinement Table 4, we compare our efficient token adaptation strategy to adding an extra post-refinement network [6] and model finetuning, including directly finetuning SAM's mask decoder or only finetuning its output token for mask prediction. Adding extra heavy post-refinement network brings limited averaged performance increase

---

Table 2: Ablation study of the HQ-Output Token on four extremely fine ane-graindd segmentation datasets. We adopt the boxes converted from their GT masks as the box prompt input. By default, we train the predicted mask of HQ Output-Tokee by computing full GT mask loss.
[35]
COIFT [29]
HRSOD [51]
ThinOhject [29]
Average
Model
mloU
mBloU
mtoU
mBleU
mloU
mBloU
mloU
mBloU
mloU
mBloU
(baseline)
62.0
92.1
86.5
90.2
73.6
Uring SAM's musk decoder feature:
SAM Context Token [5G]
71.5
62.2
93.0
87.7
91.8
85.0
84.5
73.1
85.2
77.0
SAM HQ-Ourput Token (x Ouspet Token)
75.1
65.8
93.9
88.9
93.0
86.1
86.1
74.6
87.0
78.9
IIQ Output
(Boundary Loss)
79.3
SAM
HQ-Output Token
75.3
66.0
94.2
89.2
93.0
86.1
86.8
75.4
87.3
79.2
Using Our HO-Feuttue::
SAM HIQ O0tpOu Token (+ Context Token)
78.5
70.4
94.6
89.6
93.6
87.0
88.9
79.3
88.9
81.6
SAM HQ-Output Token
78.6
70.4
94.8
90.1
93.6
86.9
89.5
79.9
89.1
81.8
Table 3: Ablation study on the HQ-Featuress sources, Early-layer denotes the feature after the first
global attention block of the ViT encoder, while final-layer denotes the output of the last ViT block.
Four HQ datasets denote DIS (val) [35], ThinObject- (test) [29], COIFT [29] and HR-SOD [51].
Fusion conv
Decoder
ViT Encoder
Four HQ datasets
Model
Mask feature
Final-layer
Early-layer
mloU
mBloU
SAM [2]]
79.5
71.1
87.3
79.2
87.8
80.1
HQ-SAM (Ours)
15.1
9.0
88.6
81.3
88.6
81.1
89.1
81.8
four HQ datasets but leads to very poor performance on COCO, indicating strong overfitting. We also observe similar phenomenon when directly finetuning SAM's mask decoder. Only finetuning SAM's output token can address the catastrophic forgetting problem with improvement on the four HQ datasets and COCO. However, the incremental improvement is still much smaller compared to ours. HQ-SAM improves 1.1 AP on COCO while output token finetuning only gives an increase of 0.4 AP g- This shows the advantage of HQ-SAM in data-efficient learning while preserving the zero-shot capability of SAM.
SAM
HQ-SAM
97.9
99.6
99.6
97.9
96.8
98.9
95.7
97.5
95.4
96.4
93.9
95.4
93.2
95.5
98.6
97.6
95.1
96.9
91.5
88.9
87.1
84.0
Recall Rate
Recall Rate
85.4
76.3
60
53.3
49.6
Thresbold
Blol
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
Threshold
BloU
0.5
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
(a) Recall comparison COIFT
(b) Recall comparison HRSOD
Figure Recall rate comparison between COIFT [29] and HRSOD [51] under the zero-shot protocol, using BIoU thresholds ranging from loose to strict. The performance gap between SAM and our HQ-SAM increases significantly when we vary from a loose BloU threshold 0.5 to a very strict threshold of 0.9, showing the advantage of HQ-SAM in predicting very accurate segmentation masks.
Accuracy analysis at different BIoU thresholds Figure 4 compares SAM and HQ-SAM from loose to strict BloU thresholds. We plot the percentage of mask predictions that have BloU larger than the threshold indicated on the x x-axis. The large performance gap with strict IoU thresholds on both COIFT [29] and HRSOD [51] clearly validates the advantage HQ-SAM in predicting very accurate masks. However, even at the loose threshold of 0.5. HQ-SAM reduces the number of incorrect predictions by SAM by for COIFT and 69% for HRSOD. This shows that HQ-SAM predictions not only substantially more accurate but also more robust in challenging cases.

---

Table Comparison with model finetuning or extra post-refinement [6]. For the COCO dataset, we use a SOTA detector FocalNet-DDi DINO [53] trained on the COCO dataset as our box prompt generator.
Four HQ datasets mloU
COCO
Model
mBloU
APB
AP
APд
APM
APs
SAM (haseline)
79.5
71.1
33.3
48.5
63.9
53.1
34.1
Training the whole SAM
38.0
12.2
0.2
5.5
Add Context Token [56]
85.2
77.0
31.9
47.2
65.1
51.2
31.9
CascadePSP Post-refinement [6]
80.9
74.6
2.8
13.4
43.4
9.4
0.0
CRM Post-refinement [37]
81.4
75.4
15.9
28.7
Finetune SAM's s decoder
87.6
79.5
9.0
19.5
45.2
15.8
4.7
Finetune SAM's output token
87.6
79.7
33.7
48.7
66.0
52.3
33.6
HQ-SAM (Ours)
89.1
81.8
34.4
49.5
66.2
53,8
33.9
Table 5: Zero-shot open-world instance segmentation results comparison on UVO [42]. We use FocalNet-DINO [53] trained on the COCO dataset as our box prompt generator. sstrict denotes the boundary region with tighter threshold.
Model
APaaiel
APilrict B76
AP
APs
AP
SAM
8.6
3.7
25.6
17.3
14.4
37.7
29.7
HQ-SAM
5.0
28.2
18.5
16.3
38.6
30.1
Table 6: Zero-shot segmentation result comparison on the test set of high-quality BIG [6] benchmark using various types of input prompts. We employ PSPNet [55] to generate the coarse mask prompt.
GT Box Prompt mloU
Mask Prompt mloU
Model
mBIoU
mBIoU
SAM
81.1 86.0
70.4
66.6
41.8
HQ-SAM
75.3
86.9
75.1
4.3 Zero-shot Comparison with SAM
We perform extensive zero-shot transfer comparisons between our HQ-SAM and SAM on bench- marks, including SGinW [58], COCO [31]. UVO [42], LVIS [14], HQ-YTVIS [20], BIG [6], COIFT [29] and HR-SOD [51], where HQ-SAM outperforms SAM without bells and whistles, demonstrating its efficacy and kept generalization ability even trained with small-scale dataset.
Results on the SGinW Benchmark Equipped with the same Grounding- DINO [32] as box prompts. we also performed experiments by replacing SAM with HQ-SAM in Grounded-SAM, and obtained the first place in the Segmentation in the Wild (SGinW) competition on the zero-shot track. Note that SGin W contains 25 zero-shot in-the-wild segmentation datasets for evaluation, and Grounded- HQ-SAM with 49.6 mean AP and outperforms Grounded --addeddd-MMM obviously using the same detector.
Zero-Shot Open-world Segmentation To evaluate the zero-shot segmentation results in the open- world environment, in Table 5, we compare SAM and our HQ-SAM on the challenging UVO [42] benchmark with diverse and dense objects mask annotations. By taking the same pre-trained object detector [53] as box prompt input, our HQ-SAM improves for 1.3 Apetrict and 2.6 A.ttiit B50 over SAM.
Zero-Shot Segmentation on High-resolution BIG Dataset In Table 6, we compare the zero-shot segmentation quality between SAM and HQ-SAM the high-resolution BIG benchmark [6] with two types of prompts, including using GT object boxes or the provided coarse masks input. HQ-SAM consistently surpasses SAM, with obvious advantages using different types of prompts, and is much more robust to coarse masks prompts with partial boundary errors (provided by PSPNet [55])-
Zero-shot Instance Segmentation COCO and LVIS In Table 7, we also evaluate HQ-SAM on the popular COCO and LVIS benchmarks respectively by feeding box prompts generated by the trained detectors of these two datasets. HQ-SAM consistently outperforms SAM by 1.1 AP3 COCO and 0.7 APrrict on LVIS, showing the improved mask quality and well prelerreeereed zero-shot segmentation ability during the HQ-SAM training process.
SGinW Benchmark Results: httpe

---

Table 7: Zero-shot instance segmentation results comparison on COCO [31] and LVISv1 [14]. For the COCO dataset, ViTDet-H [28] trained on the LVIS dataset as our box prompt generator. For SAM, we use the ViT-L backbone and box prompt. We maintain the zero-shot segmentation capability of the original SAM while improving the mask quality on the boundary region.
FocalNet-DINO [53] detector trained COCO. For LVIS, adopt
COCO
LVIS
Model
AP
AP
A
AP3
APB
AP
AP
SAM
33.3
48.5
32.1
32.8
38.5
40.9
43.6
IQQSAM
34.4
49.5
32.5
33.5
38.8
41.2
43,9
SAM
HQ-SAM
88.6
88.4
83.5
86.5
86,7
86.3
86.4
50.3
66.0
66.2
69.3
70,0
mBloU
80
83.9
83.6
mBloU
50.4
55.3
55.1
56.8
59.6
40
70
18.1
20
62.8
17.9
60 Input Point
6128
Pos.
3 Pos.
5 Pos.
5 Pos.
10 Pos.
10 Pos.
Pos.
Pos.
Pos.
Pos.
10 Pos.
10 Pos. +3 Neg.
Procptt Num
+5 Neg.
+5 Neg.
+5 Neg.
(a) Performance comparison on COIFT
(b) Performance comparison on DIS
Figure 5: Interactive segmentation results comparison using varying number of input points on the
COIFT [29] (zero-shot) and DIS [35] val set. HQ-SAM consistently outperforms SAM with various point numbers, and the relative improvement is more obvious with less prompt ambiguity.
Table 8: Zero-shot Video Instance Segmentation comparison on the test set of the very accurately labeled HQ-YTVIL [20] benchmark. We utilize pre-trained
Mask [4] on YTVIS [47] as our box prompt input while reusing its object association prediction.
Model
AP
AP
AP
APM
AP/S
AP
SAM
30.2
19.1
72.9
60.7
68.1
90.5
HQ-SAM
34.0
24.3
79.5
63.6
70.5
91.1
Point-based Interactive Segmentation Comparison To investigate the segmentation performance of HQ-SAM with interactive point prompts, in Figure 5, we compare HQ-SAM to SAM with varying numbers of input points on COIFT [29] (zero-shot) and DIS [35] val set. HQ-SAM consistently outperforms SAM with different point prompts on both two datasets. We note that the relative performance increase is more significant when the prompt contains less object ambiguity with more input points information (increasing from positive point to positive points + negative points).
Zero-shot High-quality Video Instance Segmentation Besides conducting image- based segmenta- tion evaluation, we also perform video instance segmentation results comparison on the accurately annotated HQ-YTVIS benchmark [20]. We take the pre-trained Mask2Former [4] as our video box prompts and feed it into SAM and our HQ-SAM for mask prediction. In Table 8, HQ-SAM achieves remarkable gains of 3.8 points in Tube Boundary AP and 2.9 Tube Mask APM
Visualization of HQ-Output Token In Figure 6, provide visual comparison of our HQ-Output Token vs. SAM's common output token for their cross-attention maps in the last token-to-image layer of the mask decoder. We observe that HQ-Output Token attends to the boundary and thin structure regions that are missed by the common token.
Zero-shot Visual Results Comparison In Figure 7, compare HQ-SAM SAM qualitatively in a zero-shot transfer setting, where HQ-SAM significantly promotes the mask details SAM and also improves the masks of broken holes or large portion errors by the enriched semantic context. Refer to the supplemental file for more visual comparisons.
Comparison with Adapter Tuning Strategy In Table 9, we also compare our efficient token adaptation strategy to the recent Adapter Tuning [48] and LoRA [17]. We introduce lightweight adapters to ViT layers of SAM's encoder for encoder tuning and identify that this strategy leads to overfitting and its zero-shot performance on COCO decreases from 33.3 to 29.6. This validates our design choice to freeze SAM's encoder, and mainly focus on SAM's s decoder.

---

Att. De IIQ In0-Dretgott Take
SAM's
Map
Map SAM's original Tokken
of
Image
original Tokee
Image
original Fokon
Image
H1Q-Output Token
Figure 6: Cross-attention of SAM's original token vs. HQ-Output Token in the last decoder layer. HQ-Token attends to the boundary and thin structure regions that are missed by the original token.
Figure 7: Visual results comparison between SAM (top row) vs. HQ-SAM (bottom row) in zero-shot transfer setting, given the same red box point prompt. HQ-SAM produces significantly more detailed detailed-preserving results and also addresses the mask errors with broken holes.
Table 9; Comparison Adapter Tuning [48] or using LoRA [17] in SAM's encoder using ViT-L based SAM and the same HQSeg-44K. For the COCO dataset, we use the SOTA detector FocalNet-
DINO [53] trained on the COCO dataset as our box prompt generator.
COCO
Model Params (MB)
Model
AP
AP
AP,
APM
APs
Total
Trainable
SAM
33.3
48.5
63.9
53.1
34.1
1191
SAM + LORA [17]
28.6
43.7
1192.5
1.5
SAM + Encoder Adapter [48]
29.6
44.8
63.9
47.8
29.0
1203
12.0
HQ-SAM
34.4
49.5
66.2
53.8
33.9
1196.1
5.1
Mobile Efficiency Although HQ-SAM significantly boosts SAM's mask quality with negligible overhead, it shares the heavy ViT encoder of SAM, and thus cannot achieve a real-time speed in video processing. For efficient mobile deployment,
propose Light HQ-SAM based on the tiny ViT image encoder provided by MobileSAM [52]. In Figure 2, achieving running speed of 41.2 FPS, Light HQ-SAM improves the zero-shot COCO AP of MobileSAM from 44.3 to 45.0 with negligible additional cost, i.c., 1.7MB increase in model parameters.
5
Conclusion
We propose HQ-SAM, the first high-quality zero-shot segmentation model by introducing negligible overhead to the original SAM. We propose a lightweight High-quality Output Token in HQ-SAM to replace the original SAM's output token for high-quality mask prediction. After training only on 44K accuuaae masks, HQ-SAM significantly boosts the mask prediction quality of SAM, which was trained on 1.1 billion masks. The zero-shot transfer evaluation performed on segmentation benchmarks across both image and video tasks, spanning diverse objects and scenes. Our research offers timely insights into how to leverage and extend SAM-like foundational segmentation models in data-effice Mata-efficient and computation -fffooaaioonaafo manner.
10

---

References
[1] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bemstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, al. On the opportunities and risks of foundation models. arXiv preprint urXiv:2088.07556 2021.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamers.
In NeurlPS, 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey
Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[4] Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing. Mask2former for video instance segmentation. arXiv preprint arXiv:2112. 10764, 2021.
[5] Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C. Berg, and Alexander Kirillov. Boundary loU: Improving object-centric image segmentation evaluation. In CVPR, 2021.
[6] Ho Kei Cheng. Jihoon Chung, Ku-Wing Tai, and Chi-Keung Tang. CascadePSP: Toward class-agnostic and very high-rosolution segmentation via global and local refinement. In CVPR, 2020.
[7] Ho Kei Cheng and Alexander G. Schwing. XMem: Long-term video object segmentation with an atkinson-shiffrin memory model. In ECCV, 2022.
[8] Ming-Ming Cheng, Niloy Mitra, Xiaolci Huang, Philip HS Torr, and Shi-Min Hu. Global contrast based salient region detection. TPAMI, 37(3):569-5555 2014.
[9] Daan de Geus and Gijs Dubbelman. Intra-batch supervision for panoptic segmentation high-resolution images. In WACV, 2023.
[10] Philipe Ambrozio Dias and Henry Medeiros. Semantic segmentation refinement by monte carlo region growing of high confidence detections. In ACCV, 2019.
[11] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICIRR, 2021.
[12] Amin Ghiasi. Hamid Kazemi, Eitan Borgnia. Steven Reich, Manli Shu, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein. What do vision transformers learn? visual exploration. arXiv preprint arXiv:2212.005 067277. 2022.
[13] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Y Yi Lin, Ekin D Cubuk, Qune V Le, and Barret Zoph. Simple copy-paste strong data augmentation method for instance segmentation. In CVPR, 2021. [14] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR, 2019.
[15] Stefan Hinterstoisser, Vincent Lepetit, Paul Wohlhart, and Kurt Konolige. On pre-trained image features and synthetic images for deep learning. In ECCV Workshops, 2018.
[16] Neil Houlsby. Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nip. In ICML, 2019.
[17] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhe Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Lawkrank adaptation of large language models. In ICLR, 2022.
[18] Menglin Jia, Luming Tang, Bor-Chur Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and
Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
[19] Lei Ke, Martin Danelljan, Xia Li, Yu-Wing Tai, Chi- Chi-Keung Tang, and Fisher Yu. Mask transfiner for high-quality instance segmentation
CVPR, 2022.
[20] Lei Ke. Henghui Ding, Martin Danelljan, Yu-Wing Tai. Chi-Keung Tang, and Fisher Yu. Video mask transfiner for high-quality video instance segmentation. In ECCV, 2022.
[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson. Tete Xiao, Spencer Whitehead, Alexander C. Berg. Wan-Yer Lo, Piotr Dollár, and Ross Girshick. Segment anything.
In ICCV, 2023.
[22] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In CVPR, 2020.
[23] Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge
potentials. In NeurlPS, 2011.
[24] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali. Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJC,, 1956-19999-1988 2020.
[25] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In CVPR, 2022.
[26] Xiang Li, Tianhan Wei, Yau Pun Chen, Yu-Wing Tai, and Chi Keung Tang. Fss- A 1000-class dataset for shw-shot segmentation. In CVPR, 2020.
11

---

[27] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprini arXiv:210.001900 2021.
[28] Yanghao Li, Hanzi Mao. Ross Cirshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV, 2022.
[29] Jun Hao Liew, Scott Cohen, Brian Price, Long Mai, and Jiashi Feng. Deep interactive thin object selection.
In WACV, 2021.
[30] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In CVPR, 2017.
[31] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[32] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303 05499, 2023.
[33] Qi Lu, Jason Kuen, Guo Weidong, Shen Tiancheng, Gu Jiuxiang. Li Wenbo, Jia Jiaya, Lin Zhe, and Yang Ming-Hsuan Fine-grained entity segmentation. arXiv preprint 1rxiv:2211.05721 2022.
[34] Jordi Pont-Tuset. Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Horuung and Luc Van Gool. The 2017 davis challenge on video object segmentation.
Sediv:/704.0065 2017.
[35] Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool. Highly accurate dichotomous image segmentation. In ECCV, 2022.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[37] Tiancheng Shen, Yuechen Zhang, Lu Qi, Jason Kuen, Xingyu Xie, Jianlong Wu, Zhe Lin, and Jiaya Jia. High quality segmentation for ultra high-resolution images. In CVPR, 2022.
[38] Jianping Shi, Qiong Yan, L.i Xu, and Jiaya Jia. Hicrarchical image saliency detection extended essd.
TPAMI, (3(4):717-729, 2015.
[39] Towaki Takikawa, David Acuna, Varun Jampani, and Sanja Fidler. Gated-senn: Gated shape enns for semantic segmentation. In ICCV, 2019.
[40] Chufeng Tang, Hang Chen, Xiao Li, Jianmin Li, Zhaoxiang Zhang, and Xiaolin Hu. Look closer
segment better: Boundary patch refinement for instance segmentation. CVPR, 2021.
[41] Cameron Trotter, Georgia Atkinson, Matt Sharpe, Kirsten Richardson, A Stephen McGough, Nick Wright, Ben Burville, and Per Berggren. Ndd20: A large-scale few-shot dolphin dataset for coarse and fine-grained categorisation. arXiv preprint arXiv:2005. 13359, 2020.
[42] Weiyao Wang, Matt Feiszli, Heng Wang, and Du Tran. Unidentified video objects: A benchmark for dense, open- weeld segmentation. In CVPR, 2021.
[43] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything context. arXiv preprint arXiv:2304.0222 2023.
[44] Qinrou Wen, Jirui Yang, Xue Yang, and Kewei Liang. Patchdet: Patch refinement for high quality instance segmentation. In ICLR, 2023.
[45] Yinghui Xing, Qirui Wu, De Chong, Shizhou Zhang, Guoqiang Liang, and Yanning Zhang. Class-aware visual prompt tuning for vision- pre-trained model. arXiv preprint arXiv:2208.00 08340, 2022.
[46] Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang. Saliency detection via graph-based manifold ranking. In CVPR, 2013.
[47] Linjie Yang. Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV, 2019.
[48] Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition. In ICLR, 2023.
[49] Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109.1179) 2021.
[50] Yuhui Yuan, Jingyi Xie, Xilin Chen, and Jingdong Wang. Segfix: Modcl-agnostic boundary refinement for segmentation. In ECCV, 2020.
[51] Yi Zeng, Pingping Zhang, Jianming Zhang, Zhe Lin, and Huchuan Lu. Towards high-resolution salient object detection. In ICCV, 2019.
[52] Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim. Sung-Ho Bae, Seungkyu Lee, and Choong Seon
Hong. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306. 14289, 2023.
[53] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung- Shum. Dino: Detr with improved denoising anchor hoxes for end-to-end object detection. In ICLR, 2023.
[54] Hengshuang Zhao, Xiaojuan Qi. Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Ienet for real-time semantic segmentation high-resolution images. In ECCV, 2018.
[55] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
12

---

[56] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 2022.
[57]
Ziqin Zhou, Yinjie Lei, Bowen Zhang, Linggiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot semantic segmentation. In CVPR, 2023.
[58] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai. Harkirat Behl, Jianfeng Wang. Lu Yuan, et al. Generalized decoding for pixel, image, and language. In CVPR, 2023.
[59] Xueyan Zou, Jianwei Yang, Hao Zhang. Feng Li, Linjie Li, Jianfeng Gao, and Yong Jac Lee. Segment everything everywhere all once. In NeurlPS, 2023.
13

---

Supplementary Material: Segment Anything in High Quality
In this supplementary material, Section 6 first presents the additional experimental analysis our HQ- SAM, including more zero-shot transfer comparisons to SAM on both image and video benchmarks. Then, in Section 7, we describe more details of our method implementation, including the training and inference. In Section 8, we provide further details of our constructed HQSeg-44K dataset for training HQ-SAM. Section 9, we show extensive visual results comparison between our HQ-SAM and SAM on COCO [31], DIS-test [35], HR-SOD [51], NDD20 [41]. DAVIS [34], and YTVIS [47].
6 Supplementary experiments
SAM vs. HQ-SAM on Various Backbones In Table 10, we provide comprehensive comparison between HQ-SAM and SAM using various backbones, including ViT-B, ViT-L, VIT-H and TinyViT The comparison not only includes the numerical results on the four HQ datasets and COCO validation set, but also contains the model sizes/speed/memmrry HQ-SAM consistently outperforms SAM using three different backbones, with over points increase in mBIoU on the four HQ datasets. Notably, the VIT-B based HQ-SAM significantly improves the AP COCO from 28.2 to 31.3 and AP from 44.4 to 46.7, with only 1.1% 11% increase in model parameters and negligible extra memory consumption.
Table 10: SAM HQ-SAM various VIT backbones. For the COCO dataset, we use SOTA detector DoNNO [53] trained on the COCO dataset as our box prompt generator.
Four HQ datasets mBloU
COCO
Model Params (MB)
Model
FPS
Memory
mloU
AP
AP
AP2
APM
APs
Total
Learnale
SAM-B
70.6
62.3
28.2
44.4
$7.7
48.7
32.1
358
358
10.1
5.1G
HQ-SAM-B
86.3
78.1
31.3
46.7
62.9
50.5
32.0
362.1
4.1
9.8
5.1G
SAM-L
79.5
71.1
33.3
48.5
63.9
53.1
34.1
1191
1191
5.0
7.6G
HQ-SAM-I.
89.1
81.8
34.4
49.5
66.2
53.8
33.9
1196.1
5.1
4.8
7.6G
SAM-H
75.6
68.3
34.0
48.9
64.5
53.3
34.4
2446
2446
3.5
10.3G
HQ-SAM-H
89.3
81.5
34,9
49.9
66.5
54.0
34.2
2452.1
6.1
3.4
10.3G
MobileSAM
58.8
28.6
44.3
38.6
38.6
44.8
3.7G
Light HQ-SAM
81.4
71.6
29.6
45.0
40.3
1.7
41.2
3.7G
Table 11: Results on YouTubeVIS 2019 validation set and HQ- YTVIS test set using ViT-L based SAM. We adopt the SOTA detector Mask2Former [4] trained on the YouTube 2019 dataset as video boxes prompt generator while reusing its object association prediction.
YTVIS 2019
HQ- YTVIS AP
Model
AP
AP
AP75
APr.
APM
APs
APM
SAM
51.8
82.1
55.4
65.5
52.0
34.2
30.2
60.7
HQ-SAM
53.2
82.9
58.3
66.4
53.3
33.7
34.0
63.6
Zero-shot Video Instance Segmentation Comparison Extending from Table 8 of the paper (evaluation on the HQ- YTVIS benchmark [20]), we further perform comparative analysis of zero- shot video instance segmentation results on the popular YTVIS 2019 [47] validation set. We take the pre-trained Mask2Former [4] as our video box prompts and feed them into SAM and our HQ-SAM for mask prediction. Table 11, HQ-SAM achieves consistent gains of 1.4 points in Tube Mask AP, increasing SAM's performance from 51.8 to 53.2. Interestingly, we find the AP75 improvement with a higher IoU threshold for HQ-SAM much larger than APs0, further validating the advantages of HQ-SAM in high-quality mask prediction.
14

---

Zero-shot Video Object Segmentation Comparison Besides video instance segmentation, in Table 12. we further report the comparison of video object segmentation results between HQ-SAM and SAM on DAVIS validation set in zero-shot transfer protocol. We take the pre-trained XMem as our video box prompts and feed the same prompts into SAM and HQ-SAM. HQ-SAM improves SAM the J&F from 82.0 to 83.2 and the F score from 84.9 to 86.1, where FF is for measuring the contour accuracy of the video objects.
Table 12: Results DAVIS 2017 [34] validation set using VIT-L based SAM. We adopt the SOTA model XMem [7] as our video boxes prompt generator while reusing its object association prediction.
Model
J&F
J
F
SAM HQ-SAM
82.0
79.0
84.9
83.2
80.3
86.1
Robustness to Input Box Prompts In Table 13, compare HQ-SAM SAM by adding various scales of noises to the input ground truth box prompts. In practice, we cannot expect the input box prompts provided by humans in interactive modes to be identical the ground truth (GT) boxes or extremely accurate. We follow the data augmentation code in DN-DETR [25] to add different noise scales and identify that our HQ-SAM is much mBIoU advantage improves from 10.7 to 20.5 when gradually increasing the noise scales, Note that our method is not trained with noised boxes. We also visualize such noised input case in Figure 11, where SAM is more sensitive to small box location shifts that easily happened during interactive annotation.
robust compared to SAM, where the relative
Table 13: Comparison of segmentation accuracy the four HQ datasets by adding various noise levels to the GT box prompts input.
No Noise
Noise scale 0.2 mloU
Noise scale 0.4 mIoU
Model
mIoU
mBIoU
mBIoU
mBIoU
SAM
79.5
71.1
65.7
57.1
46.4
39,8
HQ-SAM
89.1
81.8
82.8
73.4:16.3
69.9
60.3
7 Additional Implementation details
Training Details During training HQ-SAM on the composed HQSeg-44K, we fix the model parameters of the pre-trained SAM model while only making the proposed HQ-SAM learnable, including HQ-Output Token, its associated three-layer MLP and three convolutions for HQ-Features fusion. Two of them are transposed convolutions (size 2x2, stride 2) used to upscale encoder embedding size from 64x64 to 256x256 We treat the new HQ-Output Token as the fifth mask token compared to the original four mask tokens in SAM's mask decoder. During training, this new HQ-Output token of size Xx25 is concatenated with SAM's mask tokens (size of 4 2x26), iou token (size of 1 x 256) and prompt tokens (size of N Nprempt x 256) as the input to the SAM's mask decoder. For example, if the input image contains N box prompts (size Nx2x256), the final concatenated input and output shape for the 2-layer mask decoder of SAM is Nx(1+++1+2) 2556. For experiments using ViT-B, ViT-L, and ViT-H-based models on training, we adopt the same training setting, with a learning rate of 1e-3 and train our HQ-SAM for 12 epochs (learning rate drops to le-4 after 10 epochs). We supervise mask prediction of the new HQ-Output token with a combination of both BCE Loss and Dice Loss.
Implementation Details We follow the same inference pipeline of SAM but use the mask prediction from HQ-Output token as high-quality mask prediction. Table 10 reports the detailed inference speed comparison using various backbones. For www.-prootngbbbaee based evaluation, we feed SAM and our HQ-SAM with the same image/video bounding boxes and adopt the single mask output mode of SAM. For interactive segmentation comparison using a single point, we follow SAM and adopt the "center" point of Ground Truth (GT) masks, which is at a maximal value location in a mask's 's interior distance transform. For multiple-point evaluation, we randomly sample the points from the GT masks and report the averaged results with three trials.
15

---

More Details of HQSeg-44K
Data compostion of HQSeg-44K In Table 14, we provide more details our composed new training dataset HQSeg-44K which contains 44,320 extremely accurate image mask annotations, where show their annotation quality in Figure HQSeg-44K is collection of six existing image datasets including DIS [35] (train set). ThinObject-5K [29] (train set), FSS [26], ECSSD [38], MSRA-10K [8], DUT-OMRON [46] with extremely fine-grat aineggraine mask labeling, where each of them contains 7.4K mask labels average. This composed training set has no images/annotations overlapping with the zero-shot evaluation datasets adopted our paper.
Effect of HQSeg-44K In Table 15, we show the advantage of using HQSeg-44K by comparing
HQ-SAM training with 44K randomly sampled images and masks from SA-1B [21]. Using the
same efficient token learning strategy, training with SA-1B (44K) decreases the averaged mBloU on the four datasets from 71.1 to 70.1, while ours improves it from 71.1 to 81.8. This validates the effectiveness of our constructed HQSeg-44K benchmark in improving mask quality. Note that the ablation experiments Table 2, Table 3, Table 4, and Table 9 of the paper are based on the constructed HQSeg-44K.
Table 14:
Data composition of our constructed HQ-Seg-44K
Dataset
DIS [35]
Thin-Object 5k [29] FSS [26]
DUTS [46]
ECSSD [38]
MSRA- 10K [8] Total
Image Num.
3000
4748
10000
15572
1000
10000
44320
DIS
FSS-1000 Thnobjeara
ECSSD
DUT-OMRON MSRA-10K 10K
Figure 8: Visualization of annotated mask quality for randomly selected cases from the six dataset components of the HQ-Seg-44K. 7oom for better viewing the fine-grained mask details.
Zero-shot results on DIS and ThinOkjec--KK We also report zero-shot results Table 16 on DIS and ThinObject-SK by removing the training splits of either or both datasets from the training of
16

---

Table 15: Comparison of the training dataset. For the COCO dataset using ViT-L-based SAM, use a SOTA detector FocalNet-DINO [53] trained on the COCO dataset as our box prompt generator.
DIS
COIFT
HRSOD
ThinObject
Averige
Model
Dataset
mloU
mBIoU
mloU
mBIoUU
mloU
mBIoUI
mloU
mBlolU
mloU
mBBloU
SAM
SA-
90.2
83.1
HQ-SAM
SA-IB-14K
60.4
51.7
91.1
86.1
88.4
80.9
73.1
61.8
78.3
70.1
HQ-SAM
H----e---eeee Seg-44K (Ours)
78.6
70.4
94.8
90.1
93.6
86.9
89.5
79.9
89.1
Figure 9: Visual results comparison between SAM (top row) vs. HQ-SAM (bottom row) on DIS test set, given the same red box prompt. HQ-SAM produces significantly more accurate boundaries.
HQ-SAM. The improvement of HQ-SAM over SAM still substantial on DIS or ThinObject (over 10.0 points DIS-mloU and 9.0 points ThinObject-mloU)) even when the corresponding training splits are removed from training.
Table 16: Zero-shot results on DIS and ThinObject-5K by removing the training splits of either or both datasets from the training of HQ-SAM. Results not obtained zero-sho shot manner (i.e, the training split was used), are shown in parenthesis to easily compare shoo results.
Training Setting
DIS-mloU
DIS-mBloU
ThinObject-mloU
ThinObject-mBloUU
SAM (baseline)
62.0
52.8
73.6
61.8
HQ-SAM (remove
and ThinOhject)
72.9
63.1
82.7
70.7
HQ-SAM (remove DIS)
74.7
(90.1)
(80.4)
HQ-SAM (remove ThinObject)
(78.4)
(70.3)
83.3
72.1
HQ-SAM (default HQSeg-44K)
(78.6)
(70.4)
(89.5)
(79.9)
9 More Visual Results Comparison
We provide more extensive visual results comparison in Figure 9 (DIS [35] test set), Figure 10 (zero- shot setting in COCO), Figure 11 (noised box input) and Figure 12 (zero-shot setting in HRSOD [51], NDD20 [41] and web images which cover objects with various structure complexities in diverse environments. In Figure 13 and Figure 14. we provide the zero-shot video segmentation results comparison on DAVIS 2017 and YTVIS 2019 benchmarks respectively. Besides, we include the dark underwater environment in NDD20 [41] and randomly selected web images in Figure 12, showing that the zero-shot segmentation power in SAM is well preserved by HQ-SAM. In Figure 12, we also include two failure cases the rightmost two columns of the third row and bottom row, where HQ-SAM improves over SAM, but still cannot achieve fully correct mask prediction.
17

---

SAM
HQ-SAM
SAM
HQ HQ-SAM
Figure 10: Visual results comparison between SAM (top row) vs. HQ-SAM (bottom row) on COCO val set zero-shot setting, using SOTA detector FocalNet-DINO [53] trained on the COCO dataset as our box prompt generator. HQ-SAM predicts masks with higher quality than SAM with less mask artifacts.
II STPO
ADIT STPD
STZO
SAM + GT Box
SAM Noised Box
HQ-SAM GT Box
HQ-SAM Noised
Figure 11: Visual results comparison between SAM (top row) HQ-SAM (bottom row) with both the GT and noised green box prompt. HQ-SAM produces much more consistent and robust segmentation results regarding to the noises in the input boxes.
18

---

SAM
HQ-SAM
SAM
HQ-SAM
Figure 12: Visual results comparison between SAM (top row and third row) vs. HQ-SAM (second row and bottom row) in zero-shot setting, given the same yellow box or point prompt. HQ-SAM produces significantly more detailed preserving masks while fixing mask errors with broken holes. The rightmost two columns in the third row and bottom row show two failure cases of HQ-SAM in extremely dark environments or very tiny metal rods.
19

---

SAM
HQ NQ-SAM
SAM
HQ- HQ-SAM
SAM
HQ-SAM
SAM
HQ HQ-SAM
Figure 13: Visual results comparison between SAM vs. HQ-SAM on video object segmentation
benchmark DAVIS 2017 in zero-shot setting, given the same video boxes prompts generated by the pre-trained XMem [7].
20

---

SAM
HQ-SAM
SAM
HQ-SAM
56188
SAM
HQ-SAM
SAM
HQ-SAM
Figure 14: Visual results comparison between SAM vs. HQ-SAM on video instance segmentation benchmark YTVIS 2019 in zero-shot setting, given the same video boxes prompts generated by the pre-trained Fask2Former [4].
21


Reviews:
[{"review_id": "eMdhtd7UCi", "reviewer": "Reviewer_E7oL", "summary": "This paper proposes HQ-SAM, an improved SAM with higher-quality segmentation performance. HQ-SAM freezes the entire SAM model and appends additional learnable modules concurrently to SAM. With the HQ-output token and global-local feature fusion, HQ-SAM achieves better segmentation results and only costs marginal extra resources.", "questions": "None", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Additional training cost of HQ-SAM is marginal compared to SAM, but contributes to good performance boost.\n\n2. The idea of adding a new HQ-token specialized for HQ mask is interesting and reasonable to me.", "weaknesses": "1. Although the author claim the necessity of freezing the entire SAM model, it's better to verify this by ablation study, for example, unfreezing the entire SAM, or using PEFT methods (LoRA, adapters) with performance-parameter comparison.\n\n2. I'm curious about how HQ-SAM performs in PerSAM (personalization segmentation by SAM) ?\n\nPersonalize segment anything model with one shot (https://arxiv.org/pdf/2305.03048.pdf)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging our work with an interesting and reasonable design, good performance and marginal training cost. Here we respond to your insightful comments and questions.\n\n> **Q1.** Experiments on unfreezing the entire SAM, or using PEFT methods (LoRA, adapters) with performance-parameter comparison.\n\nPlease note that we already provided the experiment on unfreezing SAM’s whole mask decoder or SAM’s output token in Table 4 of the paper. We also studied using PEFT methods as adapter in Table 2 of the Supp. file. Combined with these experiment trials, we further provide a comprehensive zero-shot performance-parameter comparison by also unfreezing the entire SAM and using LoRA method as below:\n\n| Method | COCO-AP | COCO-AP^B | Total #Params(MB) | Trainable #Params(MB)| \n|----------|:----------:|:----------:|:----------:|:----------:|\n| SAM (**baseline**)    | 48.5    | 33.3    | 1191    | -    |\n| **Unfreeze the entire SAM**    | 5.5   | 0.2    | 1191    | 1191   |\n| Unfreeze SAM's mask decoder   | 19.5    | 9.0    | 1191    | 20.6    |\n| Unfreeze SAM's output token    | 48.7    | 33.7    | 1191    | 0.5    |\n| SAM + **LoRA**    | 43.7    | 28.6    | 1192.5   | 1.5    |\n| SAM + Adapter    | 44.8   | 29.6    | 1203   | 12.0   |\n| **HQ-SAM**    | **49.5**    | **34.4**    | 1196.1    | 5.1    |\n\nAmong these SAM training strategies with the same training setting, HQ-SAM achieves the best zero-shot COCO performance with marginal training cost. We will add this detailed comparison in the next revision.\n\n> **Q2.** How HQ-SAM performs in PerSAM?\n\nThank you for the suggestion. PerSAM is a concurrent arXiv work released just several days before the abstract submission deadline. PerSAM focuses on one-shot segmentation, while HQ-SAM is designed for zero-shot high-quality segmentation. Per the reviewer's suggestion, we here analyze the impact of using our HQ-SAM in PerSAM. Our HQ-SAM improves the PerSAM performance on DAVIS 2017 (Table 2 of the PerSAM paper) as shown below:\n\n\n|  PerSAM Variant  |  Backbone  |  J&F  |  J  |  F  |\n|------------|------------|------------|------------|------------|\n|   PerSAM-F     |   ViT-Huge    |   71.85    |   68.98    |   74.72    |\n|   PerSAM-F + **HQ-SAM**    |   ViT-Huge    |   **72.74**    |   **70.00**    |  **75.49**    |\n\n\nFurthermore, equipped with the same Grounding-DINO [a] detector as box prompt, we also performed experiments by replacing SAM with HQ-SAM in Grounded-SAM, and obtained the **first place** in the Segmentation in the Wild (SGinW) competition on the zero-shot track (hosted in CVPR 2023 workshop). Note that SGinW benchmark consists of **25** zero-shot in-the-wild segmentation datasets for evaluation, and Grounded-HQ-SAM outperforms Grounded-SAM obviously using the same detector. The benchmark leaderboard result is publicly available.\n\n[a] Liu, Shilong, et al. \"Grounding dino: Marrying dino with grounded pre-training for open-set object detection.\" arXiv preprint arXiv:2303.05499 (2023)."}, {"review_id": "TxEejPOBY0", "reviewer": "Reviewer_2HQS", "summary": "This paper proposes HQ-SAM that produces segmentation masks high quality while maintaining SAM's original promptable design, efficiency, and zero-shot generalizability, by only introducing minimal additional parameters. A learnable High-Quality Output Token is injected into SAM's mask decoder. Instead of only applying it on mask-decoder features, the proposed method first fuses them with early and final ViT features for improved mask details. This paper also proposes a dataset of 44K fine-grained masks from several sources. Extensive experiments and comparisons demonstrate the strong performance of the proposed HQ-SAM.", "questions": "See the weakness part.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tImproving SAM to produce more accurate masks is an important and also interesting topic, especially with acceptable finetuning cost.\n-\tThe proposed designs make sense and are justified by ablation studies.\n-\tHQ-SAM is extensively evaluated on 9 diverse segmentation datasets across different downstream tasks, where 7 out of them are evaluated in a zero-shot transfer protocol. \n", "weaknesses": "-\tAs shown in Table 3, HQ-SAM significantly boosts the performance on Four HQ datasets even if it has consistent settings with SAM (the first two rows). It indicates that training/finetuning on HQSeg-44K is important for HQ-SAM. So what would happen if SAM is also trained on HQSeg-44K dataset.\n-\tWhat will the performance be if removing the fusion conv from the last row of Table 3.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that HQ-SAM is important, interesting and effective with extensive experiments evaluation. Here we respond to your insightful comments and questions.\n\n> **Q1.** What would happen if SAM is also trained on HQSeg-44K dataset?\n\nWe provided the experiment of training SAM on HQSeg-44K in Table 4 of the paper, where we finetuned the whole mask decoder of SAM (4th row) or finetuned the output token of SAM (5th row). We find that while it will bring improvement over SAM on the four HQ datasets, it still achieves inferior performance than HQ-SAM on zero-shot COCO. Here, we also provide an experiment comparison by training the whole SAM on HQ-Seg44K as below:\n\n| Method | COCO-AP | COCO-AP^B |Four HQ Datasets mIoU | Four HQ Datasets mBIoU |\n|----------|:----------:|:----------:|:----------:|:----------:|\n| SAM (baseline)    | 48.5    | 33.3    | 79.5    | 71.1    | \n| Training the whole SAM    | 5.5    | 0.2    | 38.0    | 12.2    |\n| Training SAM's mask decoder   | 19.5    | 9.0    | 87.6    | 79.5    | \n| Training SAM's output token    | 48.7    | 33.7    | 87.6    | 79.7    | \n| **HQ-SAM**    | **49.5**    | **34.4**    | **89.1**   | **81.8**    | \n\n\nOur approach achieves the best results on both the four HQ datasets and COCO. We will include this analysis in the next revision of the paper.\n\n> **Q2.** Experiment on removing fusion conv from the last row of Table 3?\n\nWe provide the experiment on removing fusion conv from the last row of Table 3 below:\n\n|  Method  |  Fusion Conv  |  Mask Decoder Feature  |  Final-layer of ViT encoder  |  Early-layer of ViT encoder  |  Four HQ Datasets mIoU  | Four HQ Datasets mBIoU |\n|------------|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|\n|   HQ-SAM    |     |   &check;    |   &check;   |   &check;    |   88.6   |   81.1    |\n|   HQ-SAM    |   &check;   |   &check;   |   &check;    |  &check;    |   **89.1**   |   **81.8**    |\n\nAlso in this case, we find that the fusion conv benefits segmentation performance. We will update the Table 3 of paper with the above results."}, {"review_id": "MtpaWfJ3z1", "reviewer": "Reviewer_xJfb", "summary": "This paper proposes a high quality version of segments anything. It points out the issues of existing segment anything models in Figure 1 and proposes the corresponding solution. It proposes a high-quality output token and a global-local fusion module. The results are good at various open benchmarks. The visualizations are rich and impressive.", "questions": "Please see the weakness. Overall, I like this work. I will further improve the rate if the weaknesses are well-addressed, but might degrade the rate if not.", "limitations": "As stated by the author, speed might be a small concern. My suggestion is that the FPS gap would be smaller if TensorRT is applied. This is not a hard job, as the original SAM already contains a TensorRT conversion tool.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper improves the famous SAM model to a higher grade. This contribution is great and meaningful.\n\n2. The paper itself is also in a high quality. The writing is clear. The idea is direct and easy to understand.\n\n3. The results are impressive. Obvious improvement upon the SAM baseline on several benchmarks show the effectiveness of the proposed method.\n\n4. The proposed dataset is also a good contribution. This makes this area can be followed by more researchers.", "weaknesses": "1. The HQ-Output token seems the key point of the proposed method. It should have some distinguished difference to common tokens. Some feature-level visualization should be provided to show these differences. This will make the design explainable.\n\n2. The comparisons in experimental are all among the proposed and SAM. Although this work is designed improve SAM, other related woks should also be compared. For example, the work [33] is also designed for high quality segmentation. Similar to some other related works, they should be compared in at least on dataset.\n[33] High quality segmentation for ultra high-resolution images, CVPR 2022\n\n3. The Global-local Fusion model fuse the early and final layers. This design seems very related to the famous feature pyramid network (FPN) in object detection. Did the author try FPN before? FPN fuses more layers and might be a better choice. It would be helpful if the author could provide some ablation on this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our paper with great/meaningful contribution, good writing and impressive results. Here we respond to your insightful comments and questions.\n\n> **Q1.** Visualization of HQ-Output Token.\n\nWe provide visual comparison of our HQ-Output Token **vs.** SAM's common output token for their cross-attention maps in the last token-to-image layer of the mask decoder. The figure is in the attached one-page PDF rebuttal (Fig. 1). We observe that our HQ-Output Token attends to the boundary and thin structure regions that are missed by the common token. We will include this visual comparison into our final paper.\n\n> **Q2.** Comparison with work CRM [33].\n\nWe provide the comparison between SAM, the suggested work CRM [33] and our HQ-SAM below.\n\n|  Method  |  COCO-AP | COCO-AP^B | Four HQ Datasets mIoU | Four HQ Datasets mBIoU  |\n|------------|:------------:|:------------:|:------------:|:------------:|\n|   SAM (baseline)    | 48.5    | 33.3    | 79.5    | 71.1    |\n|   SAM + Cascade PSP [6]    |   13.4    |   2.8    |   80.9    |   74.6    |\n|   SAM + CRM [33]    |   28.7    |   15.9    |   81.4    |   75.4    |\n|   HQ-SAM    | **49.5**    | **34.4**    | **89.1**   | **81.8**    | \n\nOur approach achieves the best performance both on the COCO dataset and the Four HQ Datasets, where CRM [33] shows poor results when generalizing to zero-shot COCO. We will add the comparison to CRM [33] in our paper.\n\n> **Q3.** Ablation experiment on using FPN.\n\nInstead of following FPN to build a feature pyramid for fusion, our Global-local Fusion directly fuses the early-layer and final-layer features by a simple up-sampling and direct element-wise sum up. Here, we provide ablation experiments by replacing Global-local fusion by FPN, where global-local fusion obtains better results both on the COCO and four HQ datasets despite its design simplicity. FPN is designed for ResNet-based backbone, and may not be optimal for ViT based backbone with the same spatial resolution output after each block of the backbone.\n\n\n|  Method  |  COCO-AP | COCO-AP^B |Four HQ Datasets mIoU | Four HQ Datasets mBIoU  |\n|------------|:------------:|:------------:|:------------:|:------------:|\n|   HQ-SAM using **FPN**    |   49.0   |   33.8    |   87.4    |   79.0    |\n|   HQ-SAM using **Global-local Fusion**  | **49.5**    | **34.4**    | **89.1**   | **81.8**    |\n\nWe will include this ablation experiment on FPN in our paper.\n\n> **Q4.** The speed limitation.\n\nTo address the speed limitation of HQ-SAM, we replace the default Vit-B/L/H with the ViT-tiny backbone provided by MobileSAM [a], and then follow the same HQ-SAM training strategy. The resulting model, which we dub it as Light HQ-SAM, achieves 45.0 AP on COCO attaining a **real-time speed of 41.2 FPS**. Light HQ-SAM only has a 40.3MB model size, but achieves better zero-shot COCO performance than ViT-B based SAM with 358MB model size. Below we provide a detailed comparison among an array of SAM variants.\n\n|  Model  |  Inference Speed (FPS)  |  Four HQ Datasets mIoU | Four HQ Datasets mBIoU  |  COCO-AP | COCO-AP^B  |  Total #Params(MB) | Trainable #Params(MB)  |   Memory  |\n|------------|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|\n|   SAM-B    |  10.1    |  70.6    |   62.3    |   44.4    |   28.2    |   **358**    |   **358**    |     5.1G    |\n|   HQ-SAM-B    |  9.8    |  86.3    |   78.1    |   46.7    |   31.3    |   362.1    |   4.1    |     5.1G    |\n|   SAM-L    | 5.0    |   79.5    |   71.1    |   48.5    |   33.3    |   1191    |   1191    |     7.6G    |\n|   HQ-SAM-L    |  4.8    |   89.1    |   81.8    |   49.5    |   34.4    |   1196.1    |   5.1    |    7.6G    |\n|   SAM-H    |  3.5    |  75.6    |   68.3    |   48.9    |   34.0    |   2446    |   2446    |     10.3G    |\n|   HQ-SAM-H    |  3.4    |   89.3    |   81.5    |   49.9    |   34.9    |   2452.1    |   6.1    |    10.3G    |\n|   MobileSAM    |  44.8    |  69.0    |   58.8    |   44.3    |   28.6    |   38.6    |   38.6   |     3.7G    |\n|   **Light HQ-SAM**    |**41.2**    |   81.4    |   71.6    |   45.0   |   29.6    |   **40.3**    |   **1.7**    |      3.7G    |\n\nWe provide a comprehensive performance-speed-model size comparison on SAM variants in Figure 3 of the one-page PDF rebuttal. As the reviewer suggests, if TensorRT is further applied, the speed of HQ-SAM and Light HQ-SAM can be further improved. However, even without such optimizations, Light HQ-SAM achieves real-time operating speed.\n\n[a] Zhang, Chaoning, et al. \"Faster Segment Anything: Towards Lightweight SAM for Mobile Applications.\" arXiv preprint arXiv:2306.14289 (2023)."}, {"review_id": "TxxogQjbqH", "reviewer": "Reviewer_eEa5", "summary": "The authors observe that the segmentation results of SAM model are unsatisfactory in several cases such as coarse mask boundaries and increect predictions. Thus, this paper presents HQ-SAM, whose objective is to improve the segmentation capability of SAM, with the introduction of the learnable high-quality output token and the global-local fusion module. For the training process, the SAM model is freezed while the newly added components are trained on the combination of 6 datasets, i.e., DIS (train set), ThinObject-5K (train set), FSS-1000, ECSSD, MSRA177 10K and DUT-OMRON. Experiments on four datasets (DIS, COIFT, HRSOD and ThinObject) show the effectiveness of the proposed method. Nevertheless, using DIS and ThinObject as testing sets may be unfair when comparing with SAM that may not see these two datsets during training.", "questions": "1. Why do authors select these 6 datasets to form HQSeg-44K benchmark? What's the motivation behind this kind of selection? Please also see weaknesses box for other questions. I hope the authors could address my concerns in rebuttal.", "limitations": "As authors claimed, HQ-SAM shares the heavy ViT encoder of SAM, and thus cannot achieve a real-time speed in video processing.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of improving SAM model is good. SAM is a powerful foundation model in computer vision, but its performance is not always satisfactory. This paper takes a very first step towards improving SAM. \n2. Good presentation and easy to read.", "weaknesses": "1. From Fig.2, it seems that there is an error correction module, but I didn't see any descriptions about this module in Section 3.2.\n2. The training set is composed of DIS (train set), ThinObject-5K (train set), FSS-1000, ECSSD, MSRA177 10K and DUT-OMRON, while the valuations is conducted on DIS and ThinObject, which is unfair when comparing with SAM model. From Table 2, the improvement on DIS and ThinObject is much higher that on COIFT and HRSOD. This is because the corresponding training images are used in the training. Thus,  evaluatoin on such datasets cannot verity the zero-shot generation capability of the proposed approach.\n3. I encourage the authors to conduct experiments without using error correction during inference on COCO, LVIS, COIFT and HRSOD .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our paper being the first step to improve SAM, with good motivation and presentation. Here we respond to your insightful questions.\n\n> **Q1.** What’s the error correction module?\n\nError correction is not a module, but simply a direct element-wise sum between the predicted logits of the SAM (by Output Token) and predicted mask by HQ-Output Token. While this is stated in L193-L195 of paper, we agree that it is not clearly connected to the term \"Error correction\" used in Fig. 2. We will clarify this in next revision by updating Fig.2 and its caption. The element-wise sum operation has no model parameters. Obtaining high-quality mask using coarse mask as input is common in high-quality segmentation works [21, 6, 18] for mask correction. \n\n> **Q2.** The zero-shot evaluation experiments.\n\nAs stated in the abstract and L63-L64 of paper, we performed zero-shot comparison with SAM on **7 zero-shot benchmarks** including COCO, UVO, LVIS, HQ-YTVIS, BIG, COIFT and HR-SOD. In the Supp. file, we further provided zero-shot comparison with SAM on popular Youtube-VIS (Tab. 3 of the Supp.) and DAVIS (Tab. 4 of the Supp.). HQ-SAM consistently outperforms SAM in all these zero-shot evaluation experiments.\n\nIn addition, we evaluated HQ-SAM in the **SGinW** (Smentation in the Wild) competition, and obtained the **first place** on zero-shot track (hosted in CVPR 2023). Note SGinW benchmark adopts **25** zero-shot in-the-wild segmentation datasets as evaluation. Using the same Grounding-DINO detector as box prompt, Grounded-HQ-SAM outperforms Grounded-SAM obviously. The SGinW benchmark leaderboard is public.\n\nBesides, we also report zero-shot results here on DIS and ThinObject-5K by removing the training splits of either or both datasets from the training of HQ-SAM. Results not obtained in a zero-shot manner (i.e. the training split was used), are shown in parenthesis to easily compare zero-shot results. \n\n|  Training Setting  |  DIS-mIoU  |  DIS-mBIoU  | ThinObject-mIoU  |  ThinObject-mBIoU  | \n|------------|:------------:|:------------:|:------------:|:------------:|\n|   SAM (**Baseline**)    |   62.0    |   52.8    |   73.6    |   61.8    | \n|   HQ-SAM (**remove both DIS and ThinObject**)   |   72.9    |   63.1    |   82.7    |   70.7    | \n|   HQ-SAM (**remove DIS**)    |   74.7    |   66.2    |   (90.1)    |   (80.4)    | \n|   HQ-SAM (**remove ThinObject**)    |   (78.4)    |   (70.3)    |   83.3    |   72.1    | \n|   HQ-SAM (**Default HQSeg-44K**)   |   (78.6)    |   (70.4)    |   (89.5)    |   (79.9)    | \n\nThe improvement of HQ-SAM over SAM is still substantial on DIS or ThinObject (over 10.0 points on DIS-mIoU and 9.0 points on ThinObject-mIoU), even when the corresponding training splits are removed from training. We will include it in the next revision.\n\n> **Q3.** Experiments without error correction.\n\nThanks for the suggestion. While we explained error correction in Q1, here we provide results comparison on COCO, LVIS, COIFT and HRSOD before and after using error correction. Error correction improves results consistently on all datasets.\n\n|  Method  |  COCO-AP | COCO-AP^B  |  LVIS-AP  |  LVIS-AP^B  |  COIFT-mIoU  |  COIFT-mBIoU  |  HRSOD-mIoU  |  HRSOD-mBIoU  |\n|------------|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|\n|   **Before** error correction   |   48.5    |   33.3    |   43.6    |   38.5    |   92.1    |   86.5    |   90.2    |   83.1    |\n|   **After** error correction    |   49.5    |   34.4   |   43.9    |   38.8   |   94.8    |   90.1    |   93.6    |   86.9    |\n\n> **Q4.** Why selecting these 6 datasets to form HQSeg-44K benchmark?\n\nAs in L175 to L177 of paper, to our knowledge, these are six existing image datasets with highly accurate mask labels and extreme boundary details. They provide very accurate manual annotation on objects with complex structures. We show their annotation quality in the attached PDF rebuttal (Fig. 2). In Tab. 7 of the Supp. file, we analyze the effect of replacing the HQ-Seg-44k training data with the same number of masks provided in SA-1B. The results degrade severely, achieving similar performance to the original SAM.\n\n> **Q5.** The speed limitation.\n\nTo address the speed limitation, we replace the default Vit-B/L/H with the ViT-tiny backbone provided by MobileSAM, and then follow the HQ-SAM training strategy. The resulting model dubbed as Light HQ-SAM here achieves 45.0 AP on COCO,  reaching a **real-time speed of 41.2 FPS**. Light HQ-SAM only has a 40.3MB model size, but achieves better zero-shot COCO AP than ViT-B based SAM with 358MB model size. We provide a detailed comparison for SAM variants below (also in Fig. 3 of the PDF rebuttal).\n\n|  Model  |  Inference Speed (FPS)  |  Four HQ Datasets mIoU | Four HQ Datasets mBIoU  |  COCO-AP | COCO-AP^B  |  Total #Params(MB) | Trainable #Params(MB)  |   Memory (G)  |\n|------------|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|\n|   SAM-B    |  10.1    |  70.6    |   62.3    |   44.4    |   28.2    |   358    |   358    |     5.1    |\n|   HQ-SAM-B    |  9.8    |  86.3    |   78.1    |   46.7    |   31.3    |   362.1    |   4.1    |     5.1    |\n|   SAM-L    | 5.0    |   79.5    |   71.1    |   48.5    |   33.3    |   1191    |   1191    |     7.6    |\n|   HQ-SAM-L    |  4.8    |   89.1    |   81.8    |   49.5    |   34.4    |   1196.1    |   5.1    |    7.6    |\n|   SAM-H    |  3.5    |  75.6    |   68.3    |   48.9    |   34.0    |   2446    |   2446    |     10.3    |\n|   HQ-SAM-H    |  3.4    |   89.3    |   81.5    |   49.9    |   34.9    |   2452.1    |   6.1    |    10.3    |\n|   MobileSAM    |  44.8    |  69.0    |   58.8    |   44.3    |   28.6    |   38.6    |   38.6   |     3.7    |\n|   **Light HQ-SAM**    |**41.2**    |   81.4    |   71.6    |   45.0   |   29.6    |   **40.3**    |   **1.7**    |      3.7    |"}]

Author rebuttal:
We appreciate the positive and insightful comments from all four reviewers. We are glad that they found our paper "well motivated and interesting” (E7oL, 2HQS, eEa5), “important and a great contribution” (2HQS, xJfb),  “effective with extensive experiments” (E7oL, 2HQS, xJfb),  “with marginal training cost” (E7oL, 2HQS), "achieving impressive results" (xJfb), and “well written” (xJfb, eEa5). 

HQ-SAM is the first work to upgrade SAM for **high-quality** segmentation, achieving consistent performance gains on a wide range of zero-shot segmentation benchmarks with marginal cost. HQ-SAM offers timely insights into how to leverage and extend SAM-like foundational segmentation models in a data-efficient and computation-affordable manner to achieve high-quality results. Both our code and dataset are public to facilitate future research. We address all questions in separate responses to each reviewer, and will incorporate all valuable feedbacks into the our paper. 

Meta-review: This paper proposes HQ-SAM, which is an improved SAM with higher-quality segmentation accuracy. HQ-SAM freezes the SAM model and appends additional learnable modules to SAM. With the HQ-output token and global-local feature fusion, HQ-SAM achieves better segmentation results and only costs marginal extra resources.

All reviewers are positive about the paper after the rebuttal. The authors should also include the additional results presented in the rebuttal to reviewers E7oL, eEa5, xJfb in the final version.
Decision: Accept (poster)