Deep Instruction Tuning for Segment Anything Model

Authors: xiaorui  huang
gen  luo
chaoyang  zhu
bo  tong
yiyi  zhou
xiaoshuai  sun
rongrong  ji
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-27 03:09:30

Abstract: Recently, Segment Anything Model (SAM) has become a research hotspot in the fields of multimedia and computer vision, which exhibits powerful yet versatile capabilities on various (un) conditional image segmentation tasks. Although SAM can support different types of segmentation prompts, we note that, compared to point- and box-guided segmentations, it performs much worse on text-instructed tasks, e.g., referring image segmentation (RIS). In this paper, we argue that deep text instruction tuning is key to mitigate such shortcoming caused by the shallow fusion scheme in its default light-weight mask decoder. To address this issue, we propose two simple yet effective deep instruction tuning (DIT) methods for SAM, one is end-to-end and the other is layer-wise. With minimal modifications, DITs can directly transform the image encoder of SAM as a stand-alone vision-language learner in contrast to building another deep fusion branch, maximizing the benefit of its superior segmentation capability. Extensive experiments on three highly competitive benchmark datasets of RIS show that a simple end-to-end DIT can improve SAM by a large margin, while the layer-wise DIT can further boost the performance to state-of-the-art with much less data and training expenditures. Our code is released at: https://github.com/wysnzzzz/DIT.

Main paper: 
 introduction: 
 Motivated by the great success of large language models (LLMs) [59, 65] , Kirillov et al. [28] recently propose a Segment Anything project to build a foundation model for image segmentation, termed Segment Anything Model (SAM). SAM demonstrates an impressive yet versatile segmentation capacity by pre-training on over one billion masks. Moreover, it can support interactive segmentation conditioned on various input prompts [28] , such as points, boxes or texts. Its outstanding ability also arouses great interest from the communities of multimedia [4, 5, 13, 85] and computer vision [6, 8, 71] , and has been recently transferred to various segmentation tasks, such as medical image segmentation [15, 61, 76] , urban image segmentation task [88] , and marine animal segmentation [87] , or to provide instance masks for image editing tasks [13, 77] .However, the text-awareness of SAM is much inferior than following the prompts of points or boxes [28] . For instance, on truecolor segmentation task [75] , SAM can achieve state-of-the-art performance of 85.2% mIoU with box guidance. In stark contrast, SAM , the endto-end (red) and layer-wise (blue) ones. Compared with the default shallow fusion, DITs regard the visual encoder as an deep multi-modal learner for cross-modal fusion and visual feature learning, thereby achieving the full interactions between text and image for text-guided SAM. can only achieve 58.0% mIoU when fine-tuned on the RefCOCO benchmark [86] of the popular text instructed task, i.e., referring image segmentation (RIS) [86] as shown in Fig. 2 . To explain, text instructions naturally exhibit linguistic ambiguities, while location prompts (points and boxes) are precise. Meanwhile, recent Transformer-based vision models [11, 50] have well incorporated with this type of spatial information, e.g., position embedding, and it is natural for SAM to well master these location prompts. In addition, the collection of instance-wise text descriptions is more laborious, and even the public Segment Anything project also does not release the text-conditioned annotations [28] .Even worse, the shallow fusion scheme in SAM further aggravates such weakness and degrades its multi-modal ability. Concretely, for either point, box or text, input prompt is first encoded by the prompt encoder of SAM, then it is directly fed into the light-weight encoder for mask prediction. In SAM, prompts can only interact to a limited extent with visual tokens via two crossattention layers in its mask decoder [28] . Although such mechanism is well suited for point and box prompts which are relatively easy to follow, it is insufficient for SAM to infer the intentions behind texts step-by-step. Oftentimes, SAM needs to resolve linguistic ambiguity first through the attributes and relationships of the referent, then the object can be segmented using its powerful and class-agnostic segmentation ability. To this end, we argue that "Deep text instruction tuning is essential for SAM. "To validate this argument, we propose two deep instruction tuning (DIT) methods for SAM. Different from existing RIS models [7, 19, 24, 33] that build deep fusion branches upon image encoders, we focus on improving text instruction following ability of SAM without modifying its structure. This property can help DITs make full use of its superior segmentation capability while reducing the training cost to a large extent. In particular, we directly extend Figure 2 : The comparison among the default fusion scheme of SAM (with ViT-B) and our deep instruction tuning methods on RefCOCO [86] , i.e., the end-to-end and layer-wise ones, respectively 1 . In SAM, the text prompts only interactive with the visual tokens in the shallow mask decoder, which is insufficient for cross-modal fusion and text-guided segmentation. Our simple deep tuning methods can improve the performance by a large margin without significantly changing the architecture of SAM.SAM to a deep vision-language model. Enabled by the flexibility of Transformer-based encoder [11] , we convert text instruction words to input tokens and concatenate them with the visual ones for deep multi-modal fusion and feature learning, which forms the first DIT scheme in this paper, we termed end-to-end DIT (E-DIT) , as depicted in Fig. 3 . This simple method improves SAM by a large margin, e.g., +11.8% on RefCOCO val, as shown in Fig. 2 . To further enhance instruction learning, we also propose a layer-wise instruction tuning method (L-DIT) that independently projects and inserts the text words into each layer of SAM. This layer-wise method has a similar principle with soft prompt tuning [20] . The difference is that the inserted tokens are the projected text embeddings that vary across examples, instead of task-specific and parameterized prompt tokens that are static. Besides, DITs aim to project the text words onto the well-learned visual semantic space of SAM, while prompt tuning is often to help the model better adapt the downstream task without full tuning [20] .With the proposed DIT methods, we apply SAM to the popular text-guided Referring Image Segmentation (RIS) [44] task, and term the new models as DIT-SAMs. Extensive experiments are conducted on three highly competitive RIS benchmarks, namely RefCOCO [86] , RefCOCO+ [86] and RefCOCOg [56, 58] . The experimental results not only show the great advantages of DIT-SAMs over the default SAM, e.g., up to 11.6% improvement on RefCOCO testA, but also confirm the competitiveness of DIT-SAM as a stand alone RIS model. Overall, we provide a fast and feasible solution to extend SAM's modal-modal ability.To summarize, our contributions are three-fold:• We reveal the key drawbacks of SAM in terms of text-guided segmentation and argue that deep text instruction tuning is key to success.• We propose two simple yet effective deep instruction tuning (DIT) methods for SAM, which are end-to-end and layerwise, respectively. These DIT methods pave a quick and feasible way for the multi-modal extension of SAM. [17, 64, 68] , and panoptic segmentation which merges semantic and instance segmentation tasks by assigning class labels and instance identification [27, 36] . In addition, a set of advanced segmentation networks [17, 55, 64, 68, 94] are also proposed to promote the development of image segmentation. For instance, He et al. [17] propose Mask-RCNN for object instance segmentation. SOLO [67] converts instance segmentation into a single-shot classificationsolvable problem. IFR [55] is a implicit feature refinement module for high-quality instance segmentation, and TAR [94] presents the first attempt of weakly-supervised text instance segmentation by text recognition and segmentation. Recently, the Segment Anything project [28] proposes a powerful and general model for image segmentation, termed segment anything model (SAM). This project includes the creation of an unprecedentedly large segmentation dataset, so that SAM is trained with about 1 billion image-mask pairs. Via large-scale pre-training, SAM exhibits robust and versatile performance across various segmentation challenges which can also handle different types of prompts for conditional segmentation, such as point, box and text. More recently, SAM has been widely introduced to a set of segmentation-related tasks, such as low-level structural segmentation [3] , surgical scene segmentation [60] , and medical image segmentation [15, 61, 76] . However, these applications mainly make use of the interactive capability of SAM on point or box prompts, while the text-instructed application is barely explored. Moreover, SAM also works much worse with texts than the other types of prompts due to the complexity of text information and the insufficient fusion of its default shallow decoder. In this paper, we propose deep instruction tuning for the multi-modal enhancement of SAM. 
referring image segmentation: 
 Referring image segmentation (RIS) [86] is one of the most popular text-conditioned segmentation task [21, 47, 53, 54, 63, 92] , which grounds the referent with binary masks according to the given natural language expression. Early research in RIS [43, 86] typically adopts a two-stage pipeline and frames RIS as a region-text matching problem. One-stage RIS models [35, 42, 57] recently garner growing attention from both academia and industry. In these one-stage models, text features are often embedded into the vision network and directly output the mask of the referent. Inspired by the success of Transformer, various Transformer-based RIS models [18, 39, 45, 72, 79, 92] are recently proposed to improve the multi-modal reasoning ability for RIS. VLT [10] and SeqTR [92] use a encoder-decoder framework to fuse visual and language inputs into the decoder. ReSTR [24] employs two separate Transformer encoders to handle two different modalities. More recently, attempts have also been made to explore a wider range of text conditional segmentation by combining large language models (LLMs) and training with large-scale image-text pairs [30, 49, 81, 93] . For instance, UniLSeg [49] , LISA [30] and SEEM [93] focus on textinstructed open-world segmentation, which typically require massive multi-modal training data and LLMs like LLaMA-7B [65] . Compared with these advancements, we mainly focus on addressing the text awareness of SAM and only conduct quick validations on RIS benchmarks. Among these progresses, the most relevant works to our DITs are LAVT [83] and ETRIS [34] , which also only use two modality encoders for deep multi-modal fusion. However, our DITs greatly differ in both principle and designs. In particular, LAVT and ETRIS aim to achieve multi-modal interactions via crossattention-based modules between two Transformers. In contrast, the principle of DITs is to project the text words onto the semantic space of SAM's image encoder, thus achieving feature fusion and learning in one forward network. This property also makes DIT's designs more intuitve and simpler, which also exhibits obvious merits in both training costs and performance.With the ever increasing parameter size of pre-trained models [9, 11, 65] , the direct fine-tune on downstream tasks becomes prohibitively expensive. In this case, prompt tuning [14, 16, 48, 70] is proposed to use hand-craft or learnable prompt tokens to mitigate distribution shifts between pre-training and downstream tasks. The significant achievements of prompt tuning in natural language processing (NLP) have also inspired its application to computer vision and vision-language research [46, 52, 74, 90, 91] . CoOp [91] remains CLIP parameters unchanged and uses learnable vectors as soft prompts for the input text, thereby maximizing CLIP's zero-shot retrieval capabilities. To address the overfitting issue on base classes, CoCoOp [90] proposes instance adaptation by incorporating the visual features of each image into the parameterized prompts. RPO [31] enhances the robustness of VL models by masked attention to prevent the added learnable embeddings from altering the original internal representations of the model. More recently, Wu et al. [74] propose a method of approximated prompt tuning to reduce the additional cost of the inserted prompt tokens. Regarding soft prompt tuning [32, 52, 73] , whether applied deeply or shallowly, it employs learnable tokens to assist the model in adapting to downstream data distributions without the necessity for complete tuning. Although the proposed DIT methods and prompt tuning are quite similar in process, but they still hold different principles and designs. To explain, prompt tuning often adopts learnable or hand-craft tokens to brdige the gap between pre-training and downstream tasks, which are often static to tasks [20] . In contrast, DITs aim to project the text tokens onto the semantic space of SAM, which are dynamic for different examples. Conclusively, prompt tuning primarily facilitates rapid task adaptation, whereas our DITs offer a novel approach to VL reasoning for SAM. 
method 3.1 problem definition: 
 Referring image segmentation (RIS) [86] aims to segment the referred object in an image 𝐼 conditioned on the paired natural language expression 𝑇 , its objective is formulated asEQUATIONwhere F denotes the model with its parameters 𝜃 , 𝑀 ∈ R 𝐻 ×𝑊 is the ground-truth mask, and L is the canonical cross entropy loss.To achieve this task, SAM [28] first extracts visual and text features through its vision and language encoders, based on which the light-weight decoder takes as input to predict the binary masks. Thus, the model F instantiated by SAM isEQUATIONwhere F 𝑣 , F 𝑡 and F 𝑑 refer to visual backbone, language encoder and mask decoder, respectively. The mask decoder is very lightweight and only contains two cross-attention layers, which are in charge of both mask decoding and multi-modal interactions. In this case, SAM can be regarded as one of the late-fusion models which proved to be less effective than the early-fusion ones [44, 53, 54, 92] in addressing the complicated multi-modal reasoning tasks. Meanwhile, due to the lack of training on massive cross-modal data, text-guided SAM performs worse than the point-or box-guided SAM, leading to much inferior cross-modal abilities than existing SOTA methods in RIS [44, 92] .We introduce Deep Instruction Tuning (DIT) for SAM to better follow the intentions reflected in the natural language expression 𝑇 . In particular, DIT keeps the whole SAM architecture unchanged and directly uses the image encoder as a strong multi-modal fusion network. The text instruction words can continually interact with the visual contents through the whole feature learning process, helping SAM progressively locate the referent. Specifically, SAM with DIT can be then formulated asEQUATIONHere, 𝜙 (•) is a mapping function to project the text features onto the default visual space of SAM.One straightforward solution for SAM is to directly feed the instruction words to its ViT-based image encoder. Due to the flexibility of Transformer architecture, the text features can be also regarded as additional input tokens for feature transformations. This attempt is also validated in previous VL works like ViLT [25] , and has a similar process with the popular prompt tuning [20] .Table 1 : Comparison of different text-instructed schemes on RefCOCO. Default refers to the default lightweight fusion scheme of SAM. Add dec layers is to increasing the depth of the mask decoder. * denotes that the image backbone is frozen, while the text and mask decoders are tuned. The best and second best performances are in bold and underlined.Method val testA testB P@0.5 P@0.7 P@0.9 oIoU mIoU P@0.5 P@0.7 P@0.9 oIoU mIoU P@0.5 P@0.7 P@0. Here, we term this DIT solution as the end-to-end tuning, abbreviated as E-DIT. Concretely, given a text instruction 𝑇 , we first use the language encoder, i.e., BERT [9] , to extract its representations, denoted as F 𝑡 ∈ R 𝑙 ×𝑑 , where 𝑙 denotes the length of instruction. The visual features after the patch embedding layer are denoted as F 0 𝑣 ∈ R 𝑚×𝑑 . Here, 𝑚 = 𝐻 × 𝑊 /𝑃 2 is the number of patches, where 𝐻 × 𝑊 is the image resolution and 𝑃 2 is the patch size. Afterwards, we use a linear layer to project F 𝑡 onto the visual semantic space of SAM, and obtain F 0 𝑡 viaEQUATIONwhere W 0 ∈ R 𝑑 ×𝑑 and 𝑏 0 ∈ R 𝑑 denote the projection weight and the bias term, respectively. These text tokens and image patches are then processed by the following Transformer layers of the encoder for both image feature learning and cross-modal fusion. The output image features F 𝑛 𝑣 alone with the text features F 𝑡 are fed to the lightweight mask decoder. In this case, E-DIT is formulated byEQUATIONEQUATIONEQUATIONHere, 𝐿 𝑖 denotes the i-th Transformer layer of visual backbone, and 𝑊 𝑜 ∈ R 𝑑 ×𝑑 is the projection weight. [•] denotes concatenation. 𝑀 ′ is the predicted binary mask.Considering the case of freezing SAM's backbone, Eq. 4 only adds a few parameters for semantic projection. When the hidden Transformer layers are frozen, the inserted text tokens do not fit well into visual space. For the output of each layer, we also add a linear projection to facilitate text adaption.According to Wu et al. [74] , it is often difficult for additionally inserted tokens to fully interact with the existing feature sequences in the well learned semantic space, especially under the end-toend manner. In this case, we also propose an effective layer-wise instruction tuning method for SAM, demoted as L-DIT.Concretely, we project the text tokens onto the sub-semantic space of each Transformer layer of SAM' encoder, thereby mitigating the modality gap for better tuning. Given the text features F 𝑡 , we linearly insert them into each layer of the image encoder, i.e., F 𝑖 𝑡 , and the input sequence is defined byEQUATIONwhere 𝑖 denotes the 𝑖-th layer. Similar to Eq. 5-7, the final output visual tokens F 𝑘 𝑣 as well as the projected text features F ′ 𝑡 are fed to the lightweight decoder for mask prediction. Compared with E-DIT, this layer-wise scheme can better project text tokens onto the subspaces of SAM's encoder, thereby improving the efficiency of text instruction tuning. 
experiments: 
 To validate the proposed DITs, we conduct extensive experiments on three highly competitive benchmarks of RIS, i.e., RefCOCO [86] , RefCOCO+ [86] and RefCOCOg [56] , and also compare L-DIT with a bunch of state-of-the-art (SOTA) methods [40, 44, 83, 92] in RIS.RefCOCO [86] dataset is collected in an interactive two-player game [23] . There are 142,210 referring expressions and 50,000 segmentation masks in 19,994 images collected from COCO [38] . It is divided into train, val, testA, and testB with 120,624, 10,834, 5,657, and 5,059 samples, respectively. RefCOCO+ [86] contains 141,564 natural language expressions and 49,856 masks in 19,992 images. Compared to RefCOCO, expressions in RefCOCO+ describe more about attributes of the referent, e.g., color, shape, digits, and avoid using words of absolute spatial location such as left and right. RefCOCOg [56, 58] Prompt val testA testB P@0.5 P@0.7 oIoU mIoU P@0.5 P@0.7 oIoU mIoU P@0.5 P@0. Table 3 : The impact of different text word injections for L-DIT. CrossAtt and FFN refer to mutli-head cross attention and feed-forward network [66] , respectively. Linear projection is the setting of L-DIT.Setting val testA testB P@0.5 P@0.7 P@0.9 oIoU mIoU P@0.5 P@0.7 P@0.9 oIoU mIoU P@0.5 P@0.7 P@0. average while that of RefCOCO and RefCOCO+ is only 3.6 and 3.5 words. It includes object appearance and location. Metrics. We use the overall intersection-over-union (oIoU), the mean intersection-over-union (mIoU), and precision at threshold of 0.5, 0.7, and 0.9 as evaluation metrics. The oIoU is measured by the ratio of the total area of intersection to the total area of union across all test samples. The metric favors large objects over small ones. The mIoU is calculated as the average of the IoU values between predictions and the ground truth for all test samples. This measure is applied uniformly to both large and small objects.Since the text-prompt version of SAM is not publicly available, we use the default image encoder and segmentation decoder of SAM, and randomly initialize the text encoder in this paper. For visual backbone, we use ViT-B/16 initialized from pre-trained weights of SAM. BERT with 12 layers and a hidden dimension of 768 is used as language encoder. We keep the aspect ratio and resize the image to 512 × 512. Following previous works [53, 92] , we apply color augmentation, Gaussian blur and random horizontal flipping to augment the data. To binarize the prediction of RIS, we set a threshold of 0.35. The optimizer is Adam [26] with a learning rate of 1e-4, which is multiplied by a decay factor of 0.2 at the 30-th and the 35-th, and the batch size is set to 64. The warm-up epoch is configured to 3. The training process spans 40 epochs. More details can refer to our anonymously released project.4.3.1 Comparison with the default scheme. In Tab. 1, we first compare our deep instruction tuning (DIT) methods with the default fusion scheme of SAM and its alternatives, i.e., adding more layers to the mask decoder. We can see that the default fusion scheme performs much worse than DITs on RefCOCO under the setting of either fully tuning or freezing backbone, suggesting that the default lightweight decoder is insufficient in text following. With more decoding layers, this issue can be alleviated to some extent, but the performance still lags behind DITs especially when the image encoder is frozen. To explain, the additional fusion layers can enhance the cross-modal interactions, but also undermines the powerful segmentation capability of SAM pre-trained on massive data. In contrast, without modifying the main structure of SAM, our DIT methods can improve the performance greatly. Another interesting finding from Tab. 1 is that the advantage of layer-wise DIT is more obvious under the setting of freezing backbone. This result confirms our assumption that layer-wise DIT can better help to project text tokens onto the semantic space of SAM. Overall, Tab. 1 well validates the effectiveness of DITs for text-instructed SAM.Combination with prompt tuning. Tab. 2 present the results of combing DITs with conventional the prompt tuning [74, 90, 91] , i.e., adding learnable tokens to the text prompts. Under end-to-end tuning, the addition of learnable tokens can improve performance to a certain extent, mainly due to the increase of the parameter capacity of DIT for better text tuning. However, these prompt tokens are [30] ViT-H LLaVA-7B not that helpful for layer-wise DIT, on the contrary, reducing performance greatly. This result may suggests that the text words are well projected onto each layer of SAM with independent projectors. The combination with parameterized tokens somewhat declines their semantics. These results also indicate the difference between DITs and prompt tuning, i.e., one is for semantic projection while the other is for task adaption.In our layer-wise DIT, we project the text prompts directly onto the visual space of each SAM's layer. Here, we also test DIT with alternative manners in Tab. 3, such as cross-attention and cross-attention+FFN. Interestingly, these complex methods do not further enhance SAM's text-aware capability, but decline performance to some extent. This case suggests that with the strong dependency modeling of SAM's ViT encoder, a direct semantic projection can already facilitate crossmodal alignment, well confirming the motivation of our DIT.The impact of tuning on SAM. In Tab. 4, we further examine the default capability of SAM with point and box guidances on the RefCOCO benchmark. Here, we use the ground-truth boxes of RefCOCO for box-guided SAM, and randomly sample 5 points on the ground-truth mask for point-guided segmentation. From Tab. 4, we can first see that on RefCOCO, the default SAM is still likely to segment the incorrect regions of the referent with groundtruth boxes. To explain, RefCOCO is a challenging benchmark, of which images often involve dense visual semantics, and a large box will make SAM easy to segment irrelevant region. In contrast, the point sampled from the object offers a better guidance, but still lags behind SOTA performance of RIS [44, 47] . Meanwhile, the joint tuning with all types of prompts can further improve the performance of SAM on following boxes and points. We also notice that our DIT method can help SAM achieve strong text-guided performance that is already close to the box-guided capability, well validating the effectiveness of our method. In terms of the inference speed, since the number of text tokens is often about 15-20, much fewer than that of image patches, the overall impact of DITs on inference time is limited.Comparison with the State-of-the-art. In Tab. 5, we compare our L-DIT against the state-of-the-art (SOTA) methods of referring image segmentation on RefCOCO [86] , RefCOCO+ [86] , and RefCOCOg [56] datasets using the oIoU metric. Specially, on Re-fCOCO dataset, we outperforms existing SOTA methods trained from scratch with absolute improvements of 1.94%, 1.56%, 2.91% in all three splits and performs on par with the pretrained Poly-Former [44] . Furthermore, on RefCOCO+ and RefCOCOg datasets, the L-DIT 𝐻 also exhibits exceptional performance, demonstrating comparable capabilities to SOTAs [44, 84] . In addition, compared with SOTA RIS methods using a large number of visual grounding data for pre-training, Our L-DIT 𝐻 is very competitive and achieves new SOTA performance on RefCOCO val.To gain deep insights into DIT, we visualize the predictions of our DIT tuning methods and the default SAM in Fig. 4 . In the first column, it becomes evident that the default SAM struggles with a nuanced understanding of text, often yielding predictions that inadequately encapsulate the referent or outright incorrect. In contrast, the simple E-DIT offers a notable improvement of SAM's text comprehension capabilities. For instance, in the third example, SAM's segmentation exhibit a pronounced bias towards the distracting elements on the left, while E-DIT rectifies this issue, resulting in more accurate prediction. Notably, through E-DIT enhances the performance in some cases, it can lead to the loss of certain semantic information embedded within the textual features. As a consequence, E-DIT occasionally misinterpret the segmentation target, as illustrated in the last example. The last column reveals that L-DIT effectively mitigates these issues without necessitating specific designs for cross-modal features. Remarkably, even without providing explicit image positions, akin to the original SAM, L-DIT consistently delivers superior segmentation results. This demonstrates that L-DIT processing can effectively harness positional information from both modalities and maintain a strong degree of context-awareness, improving accuracy and robustness. 
conclusion: 
 In this paper, we propose two deep instruction tuning (DIT) methods to address the weakness of SAM on following text instructions, one is end-to-end and the other is layer-wise. DITs regard SAM's default visual encoder as a stand-alone multi-modal learner instead of adding additional fusion branches. Both approaches contribute to enhancing SAM's ability of segmenting target objects based on textual guidance. To validate DITs, we conduct extensive experiments on three RIS benchmark datasets. Experimental results show that our simple DIT methods can greatly improve the capability of SAM. Meanwhile, the proposed layer-wise DIT can even help SAM compete with a set of SOTA methods of RIS on all three datasets. 
 
 Appendix: The text-prompt version of SAM is not released, so we follow its setting to initialize the text encoder and tune on RefCOCO. ViT is frozen here. 


Decision: This paper proposes Deep Instruction Tuning (DIT) methods to enhance the Segment Anything Model's (SAM) capabilities in text-instructed image segmentation tasks. 

Initial concerns mainly lie on the novelty of the proposed method, and the fairness of comparisons.
The authors provided a thorough response addressing the concerns raised by the reviewers, leading to a final scores of two Borderline Reject, one Borderline Accept and one Weak Accept, with an average score of 3.75. As two reviewers still had doubts about the novelty of the paper, the AC carefully reviewed the paper and all the reviewers' comments. 

Considering the good performance and potential impact for the SAM community,the AC agrees to accept the paper and strongly recommends incorporating the content from the rebuttal into the final version.

Reviews: review: paper_summary: paper_summary: The paper targets at improving the text to mask (more formally, referring segmentation) capability of Segment Anything Model (SAM). Similar to prompt tuning, two straightforward instruction tuning methods, either end-to-end and layer wise is proposed for SAM. The novelty in design is mainly aligning the text tokens with SAM image space before conditioning the mask decoder. The experiments show effectiveness of the instruction tuning on SAM and performance improvement on major referring segmentation datasets.
strength_weakness: strength_weakness: \textbf{Simple and Effective}: Using deep instruction tuning to align text space and SAM space is a good and (somewhat incremental) novel idea to enhance SAM’s performance in referring segmentation tasks. The intuitive configurations of layer-wise and end-to-end tuning are straightforward and effective.

\textbf{Detail Experiment and Ablation} The experiments are conducted extensively with two vision backbones and evaluated on various accuracy metrics. The results are strong and comparable to previous baselines. Comprehensive ablation studies validate the contribution of each component.
limitations: limitations: The authors claim that deep text instruction tuning is essential for SAM and reports less than 60 oIOU of Default SAM on RefCOCO as a primary motivation. However, there is no official implementation of SAM with text encoding capabilities and no extra explanation in this paper. Could you elaborate more on the implementation of SAM you use? 

Using a pretrained CLIP feature without alignment could intuitively result in not good performance. Refer to some insights in
https://github.com/facebookresearch/segment-anything/issues/4

Therefore, comparing an instruction-tuned implementation with a SAM implementation that uses a pretrained CLIP encoder without additional alignment or training seems unfair. The statement that "Deep text instruction tuning is essential for SAM" is unconvincing without further evidence. In this way, these findings might be more suitable for a technical report rather than a conference paper.

I could be convinced in the rebuttal and increase the ratings if the authors address my concern towards motivation and defend research novelty well.
paper_summary: paper_summary: To enhance the text-prompt segmentation of SAM, this paper proposed two deep instruction tuning (DIT) methods. The author argued that the shallow fusion scheme in original SAM can not handle the multi-modal ability well, and it only interact to a limited extent with visual tokens via two cross-attention layers in its mask decoder. Inspired by the previous prompt tuning methods, this work introduced End-to-end DIT(E-DIT) and Layer-wise DIT(L-DIT) approaches. Extensive experiments demonstrate the effectiveness of the method and show the advantages of DIT-SAMs over SAM.
strength_weakness: strength_weakness: 1. The proposed DIT keeps the SAM architecture unchanged and directly uses the image encoder as a multi-modal fusion network.

2. DIT is simple yet effective and the experimental results show the performance advantages.

3. Providing box- and point-prompted segmentation and  the inference time comparsions with original SAM.
limitations: limitations: 1. Although it is effective compared to SAM, while the performance is not strong enough, compared to PolyFormer and LISA-7B.

2.Can the method segment multiple objects? Additional visual results would be valuable for showcasing the model's capabilities.
paper_summary: paper_summary: This paper proposes to do deep text instruction tuning for SAM. They concatenate text tokens and image tokens together to get a more accurate text-prompt segmentation result.
strength_weakness: strength_weakness: 1. Extensive experiments were conducted to fully evaluate the results of the method, including a large number of ablation studies.
2. Very well written. The article is easy to read.
limitations: limitations: 1. From Table 7, we can find that L-DIT_B cannot surpass other training from scratch methods with the same level vision backbone Swin-B. The authors need to explain why they are lagging behind other SOTA work. Or provide speed test results to prove they have an advantage at the same speed.
2. The methods are too simple and natural, and the innovation cannot meet the requirements of a top-level conference.
paper_summary: paper_summary: The paper proposes the Deep Instruction Tuning (DIT) for SAM, to improve the performance of SAM on text instruction segmentation tasks. In order to give SAM the ability to understand text prompts, this article enhances SAM into a visual-language model.
strength_weakness: strength_weakness: The method is simple and intuitive, and the paper is clearly stated.

The ablation study is comprehensive.
limitations: limitations: 1. The novelty is a bit low. The proposed method appears just to combine $F_t$ and $F_v$ for attention.

2. The proposed method does not exhibit sufficient generalization, as it is trained and tested only on the closed set. Additionally, it is necessary to demonstrate its performance in complex multi-target scenarios.

3. The proposed method does not demonstrate a significant advantage over SOTA methods, and the SOTA methods' results are also not shown in the visualizations.