EVF-SAM: Early Vision-Language Fusion For Text-Prompted Segment Anything Model

## Abstract

Abstract Segment Anything Model (SAM) has attracted widespread attention for its superior interactive segmentation capabilities with visual prompts while lacking further exploration of text prompts.
In this paper, we empirically investigate what text prompt encoders ( e . g ., CLIP or LLM) are good for adapting SAM for referring expression segmentation and introduce the E arly V ision-language F usion-based SAM ( EVF-SAM ).
EVF-SAM is a simple yet effective referring segmentation method which exploits multimodal prompts ( i . e ., image and text) and comprises a pre-trained vision-language model to generate referring prompts and a SAM for segmentation.
Surprisingly, we observe that: (1) multimodal prompts and (2) vision-language models with early fusion ( e . g ., BEIT-3) are beneficial for prompting SAM for accurate referring segmentation.
Our experiments show that the proposed EVF-SAM based on BEIT-3 can obtain state-of-the-art performance on RefCOCO/+/g for referring expression segmentation and demonstrate the superiority of prompting SAM with early vision-language fusion.
In addition, the proposed EVF-SAM with 1.32B parameters achieves remarkably higher performance while reducing nearly 82% of parameters compared to previous SAM methods based on large multimodal models.
Code and models will be made publicly available.

## 1 Introduction

Figure: Figure 1: EVF-SAM achieves competitive performance among various benchmarks for referring expression segmentation.
Refer to caption: x1.png

Segment Anything Model (SAM) (Kirillov et al., )
brings interactive segmentation paradigm to public view. Well-trained on the SA-1B dataset, SAM achieves stunning performance and quickly becomes popular as a vision foundation model for object localization and beyond. Various SAM variants (Xiong et al., ; Zhang et al., ; Zhao et al., ; Ke et al., ) have been explored, achieving better efficiency or higher precision. Despite SAM’s surprising abilities like point-prompted and box-prompted segmentation, it is a pity that the text-prompted segmentation ability remains conceptual. We retrospect such task to Referring Expression Segmentation (RES). RES focuses on the solution that one predicts the segmentation mask according to the text description given by users, which enjoys several explorations by some traditional end-to-end models (Hu et al., ; Liu et al., ; Shi et al., ; Chen et al., ; Ye et al., ; Hu et al., ; Ding et al., ; Li & Sigal, ; Wang et al., ; Yang et al., ; Liu et al., ; Wu et al., ; Liu et al., ; Yan et al., ), and is broadened by some Large Multimodal Models (LMM) (Lai et al., ; Yang et al., ; Ren et al., ; Pi et al., ; Xu et al., ; Zhang et al., ; Xia et al., ; Rasheed et al., ).
(^1^11Equal contribution.)
(^2^22Corresponding author ([xgwang@hust.edu.cn](xgwang@hust.edu.cn)).)

Figure: Figure 2: Comparisons of different Text-prompted SAM. (a) Given input texts, several works (Ren et al., ) leverage grounded detectors, e.g., Grounding DINO (Liu et al., ), to generate box prompts for SAM. (b) A natural idea to support text prompts is to use an off-the-shelf text encoder to generate text embeddings for SAM (Kirillov et al., ; Li et al., ) while the performance of referring segmentation is inferior. (c) Several works (Lai et al., ; Yang et al., ; Rasheed et al., ) adopt Large Language Models (LLM) or Large Multimodal Models (LMM) to generate prompt embeddings for SAM in an autoregressive manner, which incurs a large computation burden. (d) Our proposed EVF-SAM exploits an effective Multimodal Encoder for text-prompted SAM with higher performance and fewer parameters compared to LLM-based methods.
Refer to caption: x2.png

The key challenge lies in empowering SAM with language understanding ability for segmentation according to text prompts, e.g., referring expression segmentation.
Fig. 2 summarizes previous works which explore the text-prompted abilities of SAM:
(a) SAM with grounded detector: A two-stage framework where a grounded detector generates a bounding box to prompt SAM, e.g., Grounded-SAM (Ren et al., ). However, those methods suffer from a sub-optimal architecture, where segmentation heavily relies on the accuracy of the detector, and it is difficult to optimize due to its non-end-to-end nature.
(b) SAM with text encoder: A off-the-shelf text encoder, e.g., CLIP (Radford et al., ), is used to encode the text prompt, providing text embeddings for SAM.
Whereas the semantic gap exists between the text embeddings and SAM which is pre-trained with geometric prompts, i.e., points or boxes, thus the segmentation performance is inferior.
(c) SAM with LLM: A Large Language Model (LLM) (or Large Multimodal Model) is employed and fine-tuned to get the desired embeddings about object information. The embeddings will be used to predict segmentation masks based on image features. However, these LLM-based models are often computationally expensive, requiring massive memory and computation budgets, and the training is challenging. Additionally, complex conversation templates need to be manually designed to instruct the LLM for referring segmentation.
Can we leverage a more efficient but effective method to empower SAM with text-prompted ability in an end-to-end manner?

To this end, we empirically investigate how to encode text prompts for SAM to address referring expression segmentation.
Interestingly, we observe that (1) using multimodal prompts including both the text and image performs better than the text-only prompts and (2) the Multimodal Encoders with early vision-language fusion demonstrate significant superiority compared to text-only encoders or Large Language Models, as shown in Fig. 2 (d).

Motivated by the above observations, we extend SAM for language understanding and text-prompt capabilities by incorporating a Multimodal Encoder with Early Vision-Language Fusion (EVF) and present EVF-SAM in this paper.
The proposed EVF-SAM aims to be a simple framework to prompt SAM with texts and illustrate how to prompt SAM to follow referring expressions effectively.
EVF-SAM is built on the off-the-shelf foundation models and comprises a Multimodal Encoder, an early-fused vision-language model, e.g., BEIT-3 (Wang et al., ), and a simple projector to generate prompt embeddings for SAM.
EVF-SAM does not include elaborate designs or modules and is easy for scaling to larger models.

Training EVF-SAM is simple and conducted on referring segmentation datasets, e.g., RefCOCO (Yu et al., ), which is appropriate to adapt the original SAM for text prompts.
Despite the simple architecture, our EVF-SAM achieves superior performance on referring expression segmentation tasks and outperforms previous attempts with Large Language Models (Lai et al., ; Yang et al., ; Rasheed et al., ), as shown in Fig. 1.
The experimental results demonstrate that (1) using a multimodal encoder with the input text and image and (2) early fusion between the text and image contribute to the better-referring ability for SAM, showing a promising direction for text-prompted SAM.
Additionally, the experiments also show the superiority of our EVF-SAM using a multimodal encoder over previous methods with decoder-only Large Language Models: (1) EVF-SAM reduces huge amounts of parameters, e.g., 82% parameters compared to LISA; (2) EVF-SAM relies less on handcrafted templates or instructions, which is more efficient and flexible; (3) EVF-SAM obtains better performance with less training data.

Our main contributions can be summarized as follows:

- •
We investigate the most effective approach to prompt SAM with texts by leveraging the Multimodal Encoder with multimodal inputs and the early vision-language fusion, which outperforms vanilla text encoders or Large Language Models.
- •
We formulate the paradigm for text-prompted SAM and propose EVF-SAM, which is modular and readily integrated with mainstream foundation models. In addition, EVF-SAM gets rid of hand-crafted templates, and the training is stable and efficient compared to methods using Large Language Models.
- •
The proposed EVF-SAM, only trained with open-source datasets, achieves state-of-the-art performance on the referring expression segmentation tasks, i.e., RefCOCO/+/g, demonstrating the effectiveness of our paradigm.
Notably, EVF-SAM reduces parameters by 82% (1.3B v.s. 7.7B) compared to previous works based on Large Language Models.

## 2 Related Work

### 2.1 Text-Prompted Segment Anything Models

Segment Anything Model. SAM (Kirillov et al., ) is an interactive segmentation model capable of predicting non-semantic masks based on various types of prompts (points, boxes, coarse masks). Trained on a large-scale dataset, SAM demonstrates strong generalization capability for segmenting diverse common objects.
Several works (Xiong et al., ; Zhao et al., ) address the massive computation cost of SAM and propose efficient variants.
Efficient-SAM (Xiong et al., ) distils the image encoder of SAM, achieving comparable performance with significantly fewer parameters.
Fast-SAM (Zhao et al., ), leveraging the YOLOv8 (Jocher et al., ) architecture, achieves a $50\times$ speedup for inference.
SAM-HQ (Ke et al., ) addresses the segmentation quality of SAM and utilizes low-level features from the image encoder to enhance the mask decoder for better accuracy.
Although SAM excels in visual-based segmentation tasks with box/point/mask prompts, it currently lacks language understanding abilities and it’s infeasible to directly use text prompts for referring segmentation or semantic segmentation.

Text-Prompted explorations.
Recently, several works (Ren et al., ; Zhao et al., ; Li et al., ) have explored text prompts for SAM to segment objects according to the instructions or referring expressions.
Grounded-SAM (Ren et al., ) leverages the Grounding DINO (Liu et al., ) to obtain text-prompted boxes and feed the boxes to SAM for segmentation results, which formulates the non-end-to-end two-stage frameworks. Fast-SAM (Zhao et al., ) matches the similarity of CLIP (Radford et al., ) features between the text and Region of Interest (RoI) of image.
RefSAM (Li et al., ) employs a lightweight cross-modal MLP to project the text embeddings of the referring expressions into SAM’s sparse embeddings and dense embeddings.
LISA (Lai et al., ; Yang et al., ) employs a Large Multimodal Model, e.g., LLaVA (Liu et al., ) to extract multimodal embeddings for SAM through the auto-regressive decoder.
The aforementioned methods either suffer from poor performance or are computationally expensive. Referring expression segmentation based on SAM is a promising area for exploration, offering significant potential. We propose an effective end-to-end model that overcomes SAM’s limitations by enabling text-prompted segmentation capabilities.

### 2.2 Referring Expression Segmentation

Referring Expression Segmentation (RES) is a multimodal segmentation task requiring accurate pixel-wise segmentation and fine-grained language understanding.

Referring Segmentation via Text Encoders.
Prevalent methods (Li & Sigal, ; Wang et al., ; Yang et al., ; Liu et al., ) tend to leverage transformer-based text encoders, e.g., BERT (Devlin et al., ) or CLIP (Radford et al., ), to encode expression texts into embeddings as guidance for segmentation.
RefTr (Li & Sigal, ) uses a visual-language encoder to fuse image and text features and regresses the box and mask with a carefully designed query processor.
LAVT (Yang et al., ) leverages a hierarchical Vision Transformer (Dosovitskiy et al., ) (ViT) to perform language-aware visual encoding.
CRIS (Wang et al., ) designs a vision-language decoder to merge CLIP features, propagating fine-grained semantic information from textual representations to each pixel-level activation.
PolyFormer (Liu et al., ) follows the encoder-decoder structure, employing a transformer decoder to generate regression results.
Novel methods pay attention to being compatible with multiple tasks to formulate a uniform model. UNINEXT (Yan et al., ), UniRef++ (Wu et al., ) and UniLSeg (Liu et al., ) employ similar frameworks but focus on utilizing datasets from different fields to empower their generalization capability.
Although these traditional models are usually lightweight and achieve fine performance, They fail to integrate with large-scale foundation models, e.g., SAM(Kirillov et al., ), LLaVA(Liu et al., ), thereby struggling to keep pace with the trend of increasingly extensive pre-training.

Referring Segmentation via Large Language Models.
In the context of the rapid development of Large Multimodal Models (Liu et al., ; Bai et al., ; Sun et al., ; ) (LMM), a number of works (Lai et al., ; Yang et al., ; Ren et al., ; Pi et al., ; Xu et al., ; Zhang et al., ) have leveraged these models to encode expression texts for referring expression segmentation tasks.
LISA (Lai et al., ; Yang et al., ) finetune LLaVA (Liu et al., ) to make it able to answer questions related to segmentation with a fixed template like ‘It is [SEG].’, where the hidden embeddings at the place of special token [SEG] will be seen as multimodal features extracted by LMM.
PixelLM (Ren et al., ) extends LISA by building a segmentation codebook to enable multi-object segmentation.
PixelLLM (Xu et al., ) empowers the vision-language model to take locations (e.g., a set of points or boxes) as either inputs or outputs.
PerceptionGPT (Pi et al., ) proposes an end-to-end architecture.
u-LLaVA (Xu et al., ) supports multi-task.
PSALM (Zhang et al., ) imports mask tokens to LMM input for better performance.
However, those methods tend to adopt heavy architectures, especially the LLMs or LMMs, leading to a heavy computation burden for downstream applications.
In contrast, we find that lightweight vision-language models perform better for encode text prompts for referring image segmentation.

## 3 Method

### 3.1 Motivation: SAM with Vision-Language Models

Figure: Figure 3: Architectural explorations for text-prompted SAM. ‘L’ and ‘V’ denote the text encoder and vision encoder. We mainly explore three schemes: (a) vanilla baseline with a simple text encoder, (b) multimodal inputs with a late fusion, i.e., concatenation, and (c) multimodal inputs with early vision-language fusions, e.g., BEIT-3 (Wang et al., ).
Refer to caption: x3.png

Considering that SAM (Kirillov et al., ) has a strong generalization capability for image segmentation while the text-prompted ability has not been revealed, we investigate how to encode text prompts for SAM in this section.
We started by using the vanilla text encoder, as shown in Fig. 3 and conducted preliminary experiments on RefCOCO (testA) to evaluate the referring ability of SAM, shown in Tab. 1.

Multimodal referring information for SAM.
SAM (Kirillov et al., ) has explored the feasibility of employing a CLIP text encoder to facilitate text-prompted segmentation, as illustrated in Fig. 3 (a). We owe its weak performance to the single-modal referring information. CLIP-prompted SAM achieves 63.4 cIoU at the RefCOCO/testA benchmark, far from well-defined baselines.
CLIP exhibits strong alignment between text and image modalities, this alignment is insufficient for fine-grained tasks like segmentation. The referring information extractor should be provided with the input image and text prompts to ensure accurate alignment between the text expression and the relevant image region.
We observe performance improvements after using multimodal prompts, i.e., 63.4 v.s. 67.9 for CLIP and 65.1 v.s. 83.7 for BEIT-3.

**Table 1: Motivation analysis. Both CLIP and BEIT-3 are of Large scale, with comparable numbers of parameters. Specifically, CLIP has a total parameter count of 428M, while BEIT-3 totals 673M parameters. Metric of LLaVA (Liu et al., ) is borrowed from LISA-7B (Lai et al., )**
|  | CLIP<br>(Text) | CLIP<br>(Text+Image) | BEIT-3<br>(Text) | BEIT-3<br>(Text+Image) | LLaVA<br>(Text+Image) |
| --- | --- | --- | --- | --- | --- |
| cIoU (RefCOCO) | 63.4 | 67.9 | 65.1 | 83.7 | 79.1 |

Early-fused architecture.
Some existing works, e.g., LISA (Lai et al., ; Yang et al., ), UniLSeg (Liu et al., ), advocate for fusing visual and textual information simply before the mask generator and are widely considered as ‘early fusion’. However, we argue that these approaches are not early enough.
As illustrated in Fig. 3 (b), we define such fusion for separately encoded single-modal prompts as ‘late fusion’.
In contrast, as shown in Fig. 3 (c), we define the fusion during feature extraction, where both modalities can access the dense information of the other one, as ‘early fusion’, e.g., ViLT (Kim et al., ), BEIT-3 (Wang et al., ), which incorporate the cross-modal fusions within the encoder.

We leverage the ‘early-fusion’ vision-language model as the Multimodal Encoder to generate prompt embeddings for SAM.
Tab. 1 shows that our investigation indicates that early-fusion outperforms late-fusion, i.e., 83.7 for BEIT-3 and 67.9 for CLIP.
We believe the early-fused architecture, as defined by our approach, is beneficial for encoding text prompts since the cross-modal fusions will further enhance the semantic representation for text embeddings.
In addition, the text-to-image fusions guide the image branch to aggregate features which are aligned with text prompts, making the output embeddings more accurate for prompt SAM.

Encoder-based feature extractor.
Recently, LISA (Lai et al., ; Yang et al., ) and several LLM-based methods (Rasheed et al., ; Ren et al., ) acquire the prompt embeddings for SAM with a special token through the auto-regressive generation.
However, the uncontrolled length of the answering query introduces instability during both training and inference. Forcing the model to conform to a specific answering template can lead to language drift. In contrast, encoder-based architectures can maintain a consistent sequence length of inputs and outputs.
Utilizing the encoder-based method not only offers convenience but also yields superior performance, i.e., 79.1 for LLaVA and 83.7 for BEIT-3.
Notably, the encoder-based text-prompted SAM will reduce a massive computation burden compared to the LLM-based methods.

### 3.2 Architecture

Figure: Figure 4: The overall architecture of EVF-SAM. The proposed EVF-SAM maintains the original architecture of SAM and keeps the weights of the SAM Image Encoder frozen. EVF-SAM exploits the Multimodal Encoder with Early Vision-Language Fusion (EVF) to encode both text prompts and the low-resolution input image (which is resized to $224\times 224$). Then the output [CLS] token is projected as prompt embeddings and fed into the prompt encoder of SAM for generating the referring segmentation results.
Refer to caption: x4.png

Fig. 4 illustrates the overview of EVF-SAM, which is a simple yet effective framework with three modules: Multimodal Encoder, Projector, and Segment Anything Model (SAM).

Multimodal Encoder.
The Early Vision-Language Fused encoder adopts the input image and text and outputs fused multimodal embeddings. In EVF-SAM, we mainly adopt BEIT-3 (Wang et al., ) as the Multimodal Encoder, which formulates a multi-way transformer. The text is tokenized by XLMRobertaTokenizer (Conneau et al., ) while the image is resized to $224^{2}$ and patched by a 1/16 convolution layer. Within each block of the encoder, the image and text tokens will be fused in the attention block and then fed into separate Feed-Forward Networks (FFN). We follow ViT (Dosovitskiy et al., ) to retrieve the [CLS] token as the output multimodal embeddings.

Projector.
Different foundation models tend to have different embedding dimensions (1024 for BEIT-3-Large, 768 for BEIT-3-Base, and 256 for SAM mask decoder). We adopt a simple MLP projector containing 2 Linear layers, activated by ReLU.
In EVF-SAM, we do not design elaborate modules for better performance due to the following reasons: (1) the simple MLP is effective enough (Liu et al., ; Kim et al., ), (2) using MLP is efficient for training and inference, and (3) the simple projector will have few impacts on the pre-trained knowledge of foundation models.

Adapted prompt encoder for SAM.
SAM contains 3 main modules:
(a) Image Encoder: a Vision Transformer (Li et al., ) (ViT), extracting fine-grained feature maps from the input image.
(b) Prompt Encoder: receiving interactive prompts and encoding them into hidden embeddings.
(c) Mask Decoder: a lightweight mask generator to output the final masks based on previous embeddings.
In EVF-SAM, we maintain the architecture of the image encoder and mask decoder while extending the prompt encoder to further gather the embeddings from the Multimodal Encoder.
Specifically, the original prompt encoder encodes point or box prompts to sparse embeddings of $R^{B\times N\times D}$, where $B$, $N$, and $D$ refer to the batch size, number of points/boxes, and the embedding dimension, respectively. In EVF-SAM, the projected multimodal embeddings of $R^{B\times 1\times D}$ from the Multimodal Encoder will be concatenated to a zero-initialized sparse embeddings and then fed into the mask decoder.

### 3.3 Training

Instruction template-free. In most LLM-based frameworks, e.g., LISA (Lai et al., ; Yang et al., ), instruction templates are required to prompt the Large Multimodal Models (LMM) for the segmentation task, e.g., ‘Can you segment {object} in the picture’ with answer ‘It is [SEG].’.
Removing instruction templates will affect the performance of LMMs, therefore, users need to follow the corresponding templates for referring image segmentation.
In contrast, EVF-SAM does not require pre-training on QA (question-answering) datasets, thus eliminating the need for instruction templates. We adopt the expression phrases or sentences as input. This template-free approach simplifies training and inference.

Trainable modules.
The Multimodal Encoder (EVF) is fully trainable during our training process, allowing it to learn how to generate multimodal embeddings tailored for SAM, which requires sufficient localization information for segmentation.
For SAM, we keep the image encoder frozen during training while we enable training for the prompt encoder and mask decoder.
Our experiments revealed that freezing the prompt encoder and mask decoder only leads to a minimal performance drop while maintaining SAM’s ability. We present details in Sec. 3.

Unified training with multi-tasks.
To further enhance the generic multi-task segmentation capabilities of EVF-SAM, including semantic segmentation and fine-grained part segmentation, we present a unified training strategy for EVF-SAM with diverse training datasets, such as ADE20K (Zhou et al., ), PartImageNet (He et al., ) and PASCAL-Part (Chen et al., ).

However, we observe a performance degradation when simply mixing the training data of referring and semantic segmentation (shown in Tab. 7 of the Appendix), which can be attributed to the semantic conflict among different tasks, as discussed in UniLSeg (Liu et al., ). To alleviate the aforementioned conflicts, we leverage a special text token [semantic] and input ‘[semantic]{category}’ for semantic/part segmentation.

## 4 Experiments

### 4.1 Datasets and Metrics

Datasets.
We mainly conduct the experiments on RefCLEF (Kazemzadeh et al., ), RefCOCO, RefCOCO+ (Yu et al., ; Kazemzadeh et al., ), and RefCOCOg (Nagaraja et al., ; Mao et al., ).
Specifically, RefCOCOg contains longer expressions which are manually annotated. Except for RefCOCO+, all datasets include geometric expression (e.g., ‘on the left’). Among different splits of testing datasets, ‘testA’ is human-centric, while ‘testB’ aims for common objects.

Extra training datasets.
To further enhance the versatility of EVF-SAM, we employ multi-task unified training by expanding the training datasets by introducing Objects365 (Shao et al., ), ADE20K (Zhou et al., ), PASCAL-Part (Chen et al., ), PartImageNet (He et al., ), and HumanParsing (Liang et al., ).
Therefore, EVF-SAM can handle various granularity of text-prompted segmentation, e.g., semantic-level, instance-level, and part-level segmentation.
We refer the readers to the appendix for more details about training with the extra multi-task datasets.

Metrics.
The gIoU and the cIoU are the most commonly calculated metrics on referring expression segmentation benchmarks. The gIoU is the average intersection-over-unions (IoU) among all images in the test datasets, while the cIoU is the cumulative intersection over the cumulative union.
If not specifically declared, we follow previous works and report the cIoU as the main metric.

### 4.2 Implementation Details

Unless specified, we initialize the proposed EVF-SAM with the public weights of SAM-ViT-Huge (^1^11SAM: [https://github.com/facebookresearch/segment-anything](https://github.com/facebookresearch/segment-anything)) (Kirillov et al., ) and BEIT-3-Large (^2^22BEIT-3: [https://github.com/microsoft/unilm](https://github.com/microsoft/unilm)) (Wang et al., ).
All models are trained on 4 NVIDIA L40s GPUs with mixed precision.
We adopt DeepSpeed (Song et al., ) with ZeRO-2 for model parallel to optimize memory consumption.
During training, the batch size of each GPU is 16 and we use gradient accumulation for 2 steps, therefore the total batch size per iteration is 128.
We adopt AdamW (Loshchilov & Hutter, ) optimizer and set the initial learning rate to 1e-4 with a linear-decay schedule.
We train all models for 15k iterations (nearly 1 day) and use the binary cross-entropy loss (BCE) and dice loss (the weight of both losses is 1.0).

**Table 2: Comparison of cIoU on different benchmarks between our proposed EVF-SAM and previous state-of-the-art methods. Bold: the best results. Underline: the second-best results. AVG represents the average metric across the eight RefCOCO-series benchmarks. We abbreviate the datasets: COCO (C) (Lin et al., ), RefCOCO (RC) (Yu et al., ; Nagaraja et al., ; Mao et al., ; Kazemzadeh et al., ), Objects365 (O) (Shao et al., ), Video segmentation datasets (V), ADE20K (A) (Zhou et al., ; ), COCO-Stuff (CS) (Caesar et al., ), PACO-LVIS (PL) (Ramanathan et al., ), PASCAL-Part (PP) (Chen et al., ), GranD (G) (Rasheed et al., ), PASCAL VOC2010 (PV) (Everingham et al., ), MUSE (M) (Ren et al., ), gRefCOCO (gRC) (Liu et al., ), COCO-Interactive (CI) (Zhang et al., ), FSS-1000 (F) (Li et al., ), SA-1B (SA) (Kirillov et al., ), PartImageNet (PIN) (He et al., ), HumanParsing (HP) (Liang et al., ; ), GoldG (GG) (Kamath et al., ).**
| Method | Text Prompt<br>Encoder | SAM? | Training Data | RefCOCO | RefCOCO+ | RefCOCOg | AVG |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| val | testA | testB | val | testA | testB | val | test |  |  |  |  |  |
| LAVT ${}_{\text{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{lavt}{\@@citephrase{, }}<br>{})}}}$ | BERT-B | ✗ | RC, gRC | 72.7 | 75.8 | 68.8 | 62.1 | 68.4 | 55.1 | - | - | - |
| PolyFormer-L ${}_{\text{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{polyformer}{<br>\@@citephrase{, }}{})}}}$ | BERT-B | ✗ | RC, gRC | 76.9 | 78.5 | 74.8 | 72.2 | 75.7 | 66.7 | 71.2 | 71.2 | 73.4 |
| UNINEXT-H ${}_{\text{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{UNINEXT}{\@@citephrase{,<br>}}{})}}}$ | BERT-B | ✗ | O, C, RC, V | 82.2 | 83.4 | 81.3 | 72.5 | 76.4 | 66.2 | 74.4 | 76.4 | 76.6 |
| UniLSeg-100 ${}_{\text{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{uniseg}{\@@citephrase{,<br>}}{})}}}$ | CLIP-B | ✗ | SA, RC, gRC | 81.7 | 83.2 | 79.9 | 73.2 | 78.3 | 68.2 | - | - | - |
| UniRef++-L ${}_{\text{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{uniref++}{\@@citephrase{<br>, }}{})}}}$ | BERT-B | ✗ | RC, F, V | 79.1 | 82.1 | 77.5 | 68.4 | 74.0 | 61.5 | 71.4 | 72.8 | 73.4 |
| LISA ${}_{\text{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{lisa}{\@@citephrase{, }}<br>{})}}}$ | Vicuna-7B | ✓ | A, CS, RC, PL, PP | 74.1 | 76.5 | 71.1 | 62.4 | 67.4 | 56.5 | 66.4 | 68.5 | 67.9 |
| PixelLM ${}_{\text{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{pixellm}{\@@citephrase{,<br>}}{})}}}$ | LLaMA2-13B | ✗ | A, CS, RC, PL, M | 73.0 | 76.5 | 68.2 | 66.3 | 71.7 | 58.3 | 69.3 | 70.5 | 69.2 |
| PixelLLM ${}_{\text{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{pixelLLM}{\@@citephrase{<br>, }}{})}}}$ | T5-XL | ✓ | RC, GG | 76.9 | 78.5 | 74.4 | 69.2 | 72.1 | 64.5 | 70.7 | 72.4 | 72.3 |
| GLaMM ${}_{\text{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{glamm}{\@@citephrase{, }<br>}{})}}}$ | Vicuna-7B | ✓ | G, RC | 79.5 | 83.2 | 76.9 | 72.6 | 78.7 | 64.6 | 74.2 | 74.9 | 75.6 |
| u-LLaVA ${}_{\text{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{u-llava}{\@@citephrase{,<br>}}{})}}}$ | Vicuna-7B | ✓ | A, CS, RC, PL, PV | 80.4 | 82.7 | 77.8 | 72.2 | 76.6 | 66.8 | 74.8 | 75.6 | 75.9 |
| PSALM ${}_{\text{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{psalm}{\@@citephrase{, }<br>}{})}}}$ | Phi-1.5 | ✗ | C, RC, CI | 83.6 | 84.7 | 81.6 | 72.9 | 75.5 | 70.1 | 73.8 | 74.4 | 77.1 |
| EVF-SAM | BEIT-3 | ✓ | RC | 82.1 | 83.7 | 80.0 | 75.2 | 78.3 | 70.1 | 76.8 | 77.4 | 78.0 |
| EVF-SAM | BEIT-3 | ✓ | RC, O, A, PP, PIN, HP | 82.4 | 84.2 | 80.2 | 76.5 | 80.0 | 71.9 | 78.2 | 78.3 | 79.0 |

### 4.3 Main Results

We mainly report the cIoU metric of RefCOCO-series benchmarks and compare our proposed EVF-SAM with recent state-of-the-art methods in Tab. 2
The upper part of Tab. 2 presents traditional methods based on text encoders. Despite their advantages in terms of fewer parameters and faster inference speeds, these methods either achieve less competitive results or require vast amounts of data due to their lack of integration with foundation models. The methods listed in the lower portion of Tab. 2 are based on Large Multimodal Models (LMMs), achieving state-of-the-art (SOTA) performance but require significant computational resources.
Our EVF-SAM achieves the highest average cIoU score across all RES benchmarks, using only limited data and manageable computation costs. Specifically, our EVF-SAM achieves SOTA performance on RefCOCOg (Nagaraja et al., ; Mao et al., ), predicating a stronger capability for handling longer text prompts than previous LMM-based models, which is counter-intuitive while showing the great potential of vision-language models for understanding instructions. In addition, the early fusion between the input image and text prompts can generate more informative embeddings than independent encoders as discussed in Sec. 1.

### 4.4 Ablation Study

In this section, we conduct experiments to investigate the vision-language models for text-prompted SAM and study the effects of the designs of the proposed EVF-SAM.
Unless specified, we mainly report the cIoU on testA of RefCOCO.

**Table 3: Ablation on fusion methods. We evaluate the performance of using different pre-trained Multimodal Encoders in EVF-SAM, e.g., CLIP from OpenAI (Radford et al., ) or OpenCLIP (Ilharco et al., ). $L_{i}$ denotes the $i$-th layer in the BEIT-3 model (totally 24 layers for BEIT-3-Large). Half of the layers are activated to assess the impact of the modality fusion stage on model performance. $\dagger$: pre-trained models provided by OpenAI. $\ddagger$: pre-trained models provided by OpenCLIP.**
| Encoder | Params | Text | Image | Modality Fusion | RefCOCO | RefCOCO+ | RefCOCOg | AVG |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| val | testA | testB | val | testA | testB | val | test |  |  |  |  |  |  |
| CLIP variants. |  |  |  |  |  |  |  |  |  |  |  |  |  |
| CLIP-Large^† | 123M | ✓ |  | - | 61.0 | 63.4 | 59.9 | 43.1 | 45.9 | 40.6 | 48.9 | 49.6 | 51.6 |
| CLIP-Large^† | 428M | ✓ | ✓ | Late (Concat) | 67.4 | 68.9 | 64.4 | 50.5 | 54.6 | 46.7 | 55.1 | 56.2 | 58.0 |
| CLIP-Large^‡ | 123M | ✓ |  | - | 60.8 | 63.2 | 59.0 | 42.9 | 46.4 | 39.2 | 49.2 | 50.5 | 51.4 |
| CLIP-Large^‡ | 428M | ✓ | ✓ | Late (Concat) | 66.1 | 67.8 | 63.1 | 49.8 | 51.9 | 44.1 | 54.1 | 55.0 | 56.5 |
| CLIP-Huge^‡ | 302M | ✓ |  | - | 61.7 | 64.2 | 60.1 | 44.2 | 47.8 | 40.2 | 49.6 | 50.9 | 52.3 |
| CLIP-Huge^‡ | 986M | ✓ | ✓ | Late (Concat) | 66.3 | 68.2 | 64.3 | 49.8 | 53.5 | 45.1 | 55.4 | 56.7 | 57.4 |
| Early-fused vision-language models. |  |  |  |  |  |  |  |  |  |  |  |  |  |
| ViLT | 133M | ✓ |  | - | 61.0 | 63.0 | 60.0 | 42.5 | 45.4 | 39.5 | 49.3 | 49.5 | 51.3 |
| ViLT | 136M | ✓ | ✓ | Late (Concat) | 61.4 | 64.0 | 59.6 | 42.8 | 46.4 | 40.1 | 49.5 | 50.0 | 51.7 |
| ViLT | 136M | ✓ | ✓ | Early | 73.9 | 75.3 | 70.9 | 61.1 | 64.4 | 55.2 | 65.1 | 66.8 | 66.6 |
| BEIT-3-Large | 370M | ✓ |  | - | 61.6 | 65.1 | 59.4 | 44.0 | 47.6 | 40.6 | 49.5 | 50.8 | 52.3 |
| BEIT-3-Large | 673M | ✓ | ✓ | Late (Concat) | 67.7 | 70.2 | 65.4 | 51.1 | 55.0 | 46.9 | 57.2 | 57.0 | 58.8 |
| BEIT-3-Large | 673M | ✓ | ✓ | Early ($L_{1}\sim L_{12}$) | 80.6 | 82.2 | 78.8 | 72.4 | 75.7 | 66.7 | 73.7 | 75.0 | 75.6 |
| BEIT-3-Large | 673M | ✓ | ✓ | Early ($L_{1}\sim L_{24}$) | 82.1 | 83.7 | 80.0 | 75.2 | 78.3 | 70.1 | 76.8 | 77.4 | 78.0 |

Multimodal Encoder and fusion methods.
In Tab. 3, we explore the effects of different Multimodal Encoders, e.g., CLIP, ViLT (Kim et al., ), and BEIT-3, and fusion methods, e.g., late fusion or early fusion.
As shown in Tab. 3, using a text-only encoder in EVF-SAM obtains limited segmentation performance on RefCOCO.
Using Multimodal Encoders with both image and text inputs remarkably improves 4.5 cIoU, 4.6 cIoU, 4.0 cIoU, 1.0 cIoU, and 4.5 cIoU for CLIP-Large^† (OpenAI (^3^33OpenAI: [https://github.com/openai/CLIP](https://github.com/openai/CLIP))), CLIP-Large^‡ (OpenCLIP (^4^44OpenCLIP: [https://github.com/mlfoundations/open_clip](https://github.com/mlfoundations/open_clip))), CLIP-Huge^‡ (OpenCLIP), ViLT, and BEIT-3, respectively.
It demonstrates the superiority of using multimodal prompts (text and input image) and showcases that the image embeddings will also provide useful guidance for SAM to segment objects accurately.
We further evaluate the effects of early fusion on ViLT and BEIT-3, which adopts modality fusions in all self-attention layers.
Specifically, we adopt two settings for BEIT-3 to analyze, e.g., fusions among former 12 layers ($L_{1}\sim L_{12}$), and fusions among all layers ($L_{1}\sim L_{24}$).
Tab. 3 indicates that BEIT-3 with early fusion (fusing former 12 layers or fusing all 24 layers) significantly improves compared to late fusion or using text only.
In addition, ViLT with early fusion also achieves 11.1 cIoU improvements compared to the baseline with text-only prompts, showing the effectiveness of early fusion and multimodal inputs for prompting SAM.
Therefore, Tab. 3 demonstrates that (1) Multimodal Encoder with the input image and text and (2) early fusions between the image and text encoder are much effective for text-prompted SAM.

Ablations on trainable modules.
In Tab. 5, we evaluated the effects of fine-tuning (✓) or freezing ($\ast$) modules in the proposed EVF-SAM, i.e., the Multimodal Encoder, the prompt encoder, and the mask decoder.
The image encoder of SAM is kept frozen during training.
As Tab. 5 shows, fine-tuning the Multimodal Encoder is crucial and it adapts the Multimodal Encoder to encode text and image inputs to multimodal representation for referring image segmentation.
Notably, EVF-SAM can achieve competitive results with all modules of SAM kept frozen, and it can be seamlessly regarded as a strong extension for the original SAM, which simultaneously supports text prompts, box prompts and point prompts.
Tab. 5 Further fine-tuning the prompt encoder and mask decoder of SAM brings significant improvements.

**Table 4: Ablations on trainable modules. We mainly evaluate the effects of fine-tuning or freezing the Multimodal Encoder, the prompt encoder and mask decoder of SAM. ‘✓’ denotes trainable, while ‘$\ast$’ denotes frozen.**
| Multimodal Enc. | Prompt Enc. | Mask Dec. | cIoU |
| --- | --- | --- | --- |
| $\ast$ | $\ast$ | ✓ | 21.2 |
| ✓ | $\ast$ | $\ast$ | 82.9 |
| ✓ | $\ast$ | ✓ | 83.3 |
| ✓ | ✓ | ✓ | 83.7 |

Multimodal feature representation.
In Tab. 5, we explore the effects of using different multimodal features representations as prompts for SAM.
Specifically, we adopt different outputs of the Multimodal Encoder: (a) the image [CLS] token, (b) the AvgPool over image tokens, and (c) the text [CLS] token.
Tab. 5 shows that using image [CLS] token is more effective while combining image and text tokens through concatenation leads to a performance drop.

Effects of Different Foundation Models.
In Tab. 6, we explore the effects of using different foundation models in EVF-SAM.
For the Multimodal Encoder, we adopt CLIP-Large (only text encoder), ViLT, BEIT-3-Large, and BEIT-3-Base.
We also modify EVF-SAM with Efficient-SAM (Xiong et al., ) to formulate a lighter version, which reduces 600M parameters compared to SAM-H.
As shown in Tab. 6, EVF-SAM with BEIT-3-Base brings a severe performance drop which indicates a better Multimodal Encoder leads to better prompts for SAM.
Remarkably, Tab. 6 shows a negligible difference between Efficient-SAM-S and SAM-H in EVF-SAM, which demonstrates the effectiveness of Efficient-SAM and also indicates that EVF-SAM performs well for different SAM variants.
In addition, it also provides insights about designing text-prompted SAMs for future research, e.g., developing a larger and better Multimodal Encoder is more important to empower SAM with text-prompted abilities.

**Table 6: Comparison of effects of different foundation models. AVG represents the average metric across the 8 RefCOCO-series benchmarks.**
| Multimodal Encoder | SAM | Params | RefCOCO | RefCOCO+ | RefCOCOg | AVG |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| val | testA | testB | val | testA | testB | val | test |  |  |  |  |
| CLIP-Large | SAM-ViT-H | 1.08B | 61.0 | 63.4 | 59.9 | 43.1 | 45.9 | 40.6 | 48.9 | 49.6 | 51.6 |
| ViLT | SAM-ViT-H | 783M | 73.9 | 75.3 | 70.9 | 61.1 | 64.4 | 55.2 | 65.1 | 66.8 | 66.6 |
| BEIT-3-Base | SAM-ViT-H | 863M | 78.9 | 80.6 | 75.3 | 69.8 | 74.2 | 63.0 | 71.6 | 72.9 | 73.3 |
| BEIT-3-Large | Efficient-SAM-S | 700M | 82.5 | 83.5 | 80.4 | 75.4 | 77.9 | 70.2 | 76.1 | 77.1 | 77.9 |
| BEIT-3-Large | SAM-ViT-H | 1.32B | 82.1 | 83.7 | 80.0 | 75.2 | 78.3 | 70.1 | 76.8 | 77.4 | 78.0 |
| BEIT-3-Large | SAM-2-L | 898M | 82.7 | 84.1 | 80.0 | 76.3 | 80.1 | 71.8 | 77.0 | 78.4 | 78.8 |

### 4.5 Discussions

To unveil how the multimodal encoder contributes to prompting SAM with texts, we visualize the attention maps between the [CLS] token (prompt embeddings) and the image tokens from the last layer of BEIT-3.
As shown in Fig. 5, the attention maps focus on the target objects and are consistent with the input text prompts.
The deep fusion of text and image embeddings leads to accurate region-text alignment.
Consequently, the prompt embeddings contain abundant object-related information, including semantics and spatial localization, which is conducive to SAM achieving precise object segmentation.

Figure: Figure 5: Visualizations of Attention Maps in Multimodal Encoder. To unveil the effects of the Multimodal Encoder, we visualize the attention maps between the [CLS] token and image tokens in the last layer of BEIT-3-Large. Specifically, we sum up the attention maps from all heads.
Refer to caption: x5.png

## 5 Conclusion

In this paper, we have explored the effective ways to prompt SAM with texts and demonstrate the importance of using the Multimodal Encoder with early fusion and multimodal inputs, i.e., text prompts and input images.
To this end, we propose EVF-SAM, which establishes a new and simple path for extending SAMs’ text-prompted segmentation abilities with the off-the-shelf foundation models.
We conduct experiments on the referring expression segmentation (RES) tasks with various benchmarks to evaluate the performance of text-prompted SAM.
Experimental results showcase that our EVF-SAM achieves state-of-the-art performance for segmenting objects with referring texts on RefCOCO/+/g benchmarks, outperforming recent approaches based on Large Language Models with huge numbers of parameters.
Moreover, experiments prove that (1) a multimodal encoder with input text and image and (2) the early fusion between image and text do matter more for prompting SAM than vanilla text encoders or Large Language Models.
We hope this study and experiments can bring new ideas or insights to inspire future research on prompting SAM with texts.

## Acknowledgments

This work was partially supported by the National Science and Technology Major Project under Grant No. 2023YFF0905400 and the National Natural Science Foundation of China (NSFC) under Grant No. 62276108. We thank Bo Jiang for his helpful feedback on the draft.

## Appendix A Appendix

### A.1 Qualitative Results

In this section, we mainly visualize the qualitative results on RefCOCO val and RefCOCOg val datasets, as shown in Fig. 6 and Fig. 7, respectively.
Moreover, we compare the qualitative results of different ways to prompt SAM with texts: (1) our proposed EVF-SAM, (2) SAM with LLM (LISA (Lai et al., )), and (3) SAM with a CLIP text encoder implemented in this paper (suggested by (Kirillov et al., ), which are based on the same SAM-Huge model.
The qualitative results can demonstrate the superiority of the proposed EVF-SAM.

Visualizations on RefCOCO.
Fig. 6 shows the qualitative comparisons on the RefCOCO val, which contains simple descriptive expression texts.
The proposed EVF-SAM can follow the expressions and segment more accurately with clear boundaries.

Figure: Figure 6: Visualization Results on RefCOCO val. We compared the qualitative results on RefCOCO which contains simple descriptive expressions.
Refer to caption: x6.png

Visualizations on RefCOCOg.
Fig. 7 illustrates the qualitative comparisons on the RefCOCOg val, which aims to segment objects with long expression texts.
The SAM with a vanilla CLIP text encoder produces inferior segmentation results given the long-expression texts. However, the proposed EVF-SAM outperforms LISA when using long expressions, even though LISA
adopts LLaMA-7B (Touvron et al., ) to understand the instructions and generate prompt embeddings, showcasing that the lightweight vision-language models can understand complex expressions.
In addition, the proposed EVF-SAM can also understand the texts or expressions towards spatial locations, such as ‘the umbrella closest to the camera’.

Figure: Figure 7: Visualization Results on RefCOCOg val. Considering that RefCOCOg contains longer expressions and we provide qualitative results to show the capability of our EVF-SAM for understanding long expressions.
Refer to caption: x7.png

### A.2 Training EVF-SAM with Multi-Tasks

To further enhance the generic capability of our EVF-SAM, we propose to implement multi-task training. Based on the experiments that show the performance degradation when simply including extra segmentation data, we explore ways to make our EVF-SAM gain from extra data.

Mixed training with semantic segmentation.
We introduce some extra semantic segmentation datasets (ADE20K (Zhou et al., ), Mapillary (Neuhold et al., )) to proceed with joint training. We do not include COCO-Stuff (Caesar et al., ) to avoid data leakage with RefCOCO/+/g.
It can be seen in Tab. 7 that the performance on RefCOCO+ and ADE20K gains, indicating the effectiveness of including extra data to enhance the generic capability. However, the evaluation metrics of RefCOCO and RefCOCOg decrease when simply including extra semantic segmentation data. We owe this phenomenon to semantic conflict (Liu et al., ).

**Table 7: Results of adding extra semantic data. ^∗ means zero-shot results. The reported ADE20K results are evaluated on the validation set using the cIoU metric.**
| ADE20K | Mapillary | RefCOCO | RefCOCO+ | RefCOCOg | ADE20K |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| val | testA | testB | val | testA | testB | val | test | val |  |  |
|  |  | 82.1 | 83.7 | 80.0 | 75.2 | 78.3 | 70.1 | 76.8 | 77.4 | 54.2^∗ |
| ✓ |  | 81.7 | 83.6 | 80.3 | 75.4 | 78.4 | 71.3 | 75.5 | 77.6 | 75.9 |
|  | ✓ | 81.9 | 83.5 | 80.3 | 75.1 | 78.0 | 70.8 | 75.3 | 77.4 | 59.6^∗ |
| ✓ | ✓ | 81.8 | 83.4 | 79.7 | 75.6 | 78.0 | 70.7 | 75.8 | 76.9 | 76.1 |

Unified training with multi-task datasets.
To solve the semantic conflict mentioned above, we propose several pre-process strategies for datasets of different distributions. We will open-source related codes in our project page.

- Instance-level data:
We apply Objects365 (Shao et al., ) to extend RES data. Specifically, (a) for each image, we exclude categories with more than one instance to avoid ambiguity problem. (b) we employ SAM-2 (Ravi et al., ) to automatically annotate masks according to the selected ground-truth bounding boxes.
The remaining annotations maintain a rich amount thanks to the dense annotation of Objects365 (Shao et al., ). We obtain 524K images (of original 600K images) with 1.8M annotations (of original 10M annotations). The mask quality from automatic annotation is fine thanks to the accurate ground-truth from Objects365 (Shao et al., ) and the powerful segmentation capability of SAM-2 Ravi et al. (). Besides, the remaining annotations are valuable for addressing long-tail problems because those excluded annotations often belong to head categories.

- Semantic-level data:
We introduce ADE20K (Zhou et al., ; ) to broaden multi-task capability. We construct a special token ‘[semantic]’ and input ‘[semantic] {category}’. The special token would not be limited to common grammar so it is helpful to avoid semantic conflict.

- Part-level data:
To enable the model to segment parts of objects, we introduce PartImageNet (He et al., ), HumanParsing (Liang et al., ; ) and PASCAL-Part (Chen et al., ) to train our model.
For semantic-level annotated datasets, i.e., HumanParsing, we implement the same strategy as ADE20K. Exceptionally, we align the definition of ‘left’ and ‘right’ with RES datasets (e.g., RefCOCO).
For instance-level annotated datasets, i.e., PartImageNet and PASCAL-Part, we merge instance masks of the same category to convert the dataset to semantic-level. Then, the same strategy as ADE20K is implemented.

By combining those datasets, we observe a significant performance gain of 1.0 cIoU on the average metric, as shown in Tab. 2. Moreover, our model is able to proceed with multiple tasks like part-segmentation and semantic-level segmentation.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: More results comparison on different datasets and a detailed difference discussion to existing language grounding methods on early-fusion will make the paper's contribution more clear.
soundness: 2
strengths: 1. The paper is written in a clear structure and organized well. The figures have a nice summarization over existing text-prompt manners (fig1) and the models's architecture (fig 4).

2. The paper has a good ablation experiment analysis. Table 3 provides a comprehensive ablation experiment on various fusion methods, which shows the advantage brought by early fusion for vision-language models and supports paper's main claim. Table 4, 5 and 6 provide answers for trainable modules, feature representation and foundation model design choice.

3. The paper shows a strong result on RefCOCO, RefCOCO+ and RefCOCOg as in Figure 1.
summary: The paper proposes EVF-SAM for referring segmentation on text-prompt based segment anything model. EVF_SAM explores both multimodal prompts and vision-language models with early fusion to prompt SAM. The papers compared the early fusion to the late modality fusion manners. Experiments are conducted on RefCOCO, RefCOCO+ and RefCOCOg to show its higher performance and fewer model parameters.
weaknesses: 1. The paper has a limited tech novelty and contribution. Early cross-modal fusion between text and image features using cross-attention has been widely explored in popular language grounding methods, such as Grounding DINO, Grounding DINO1.5 [a] and GLIP [b]. There is no clear discussion on the differences to these existing methods. The language grounding tasks and referring segmentation tasks are very close related. 

2. The paper only evaluates on the COCO related datasets, more in-the-wild testing on more diverse datasets will also be appreciated, such as using the ReasonSeg proposed by LISA or ReasonSeg-Ins by LISA++.

3. In Table 2, all comparing method are using different text-prompt based encoders, this makes the results comparison not that fair. As BEIT-3 is not the contribution of the paper. If using ViLT as the encoder, EVF-SAM's model performance will decrease from 82.1 to 73.9 on Ref-COCO val set. This means the SOTA performance of EVF-SAM is highly dependent on pretrained BEIT-3 while the comparing methods in Table 2 didnot use it.

4. Besides model parameter comparison, a comparison on the running inference speed in Table 6 will help readers understand better.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: The questions raised in this section are the same as the weaknesses outlined above.
soundness: 2
strengths: 1.	The paper presents a approach to enhancing SAM’s text-prompted segmentation through EVF-SAM, which leverages multimodal encoding and early vision-language fusion. The originality lies in tackling the relatively unexplored area of text-guided segmentation in SAM, addressing significant limitations in existing grounded detectors and large language models for this task. By utilizing an encoder-based approach with early fusion, the method efficiently merges text and image modalities, ensuring a more cohesive representation and improving SAM’s capability for referring expression segmentation without the computational burden of LLMs or the need for manual prompt design.
2.	In terms of quality, the proposed method is rigorously evaluated on multiple RES datasets (RefCOCO, RefCOCO+, RefCOCOg), where it demonstrates state-of-the-art performance with a highly efficient architecture, achieving notable gains with a reduced parameter count of 1.32B, approximately 82% fewer parameters than comparable methods. 
3.	Finally, the significance of this work is good. By demonstrating a feasible, efficient way to incorporate text prompts in SAM through multimodal encoding and early fusion, the paper not only advances SAM’s application in multimodal tasks but also contributes valuable insights to the broader field of vision-language fusion.
summary: This paper proposes EVF-SAM, a novel approach to enhance SAM’s text-based segmentation capabilities by leveraging multimodal prompts and early vision-language fusion. EVF-SAM integrates a multimodal encoder (e.g., BEIT-3) with SAM through a simple projector, transforming text and image inputs into a unified embedding optimized for segmentation tasks. Using instruction-free templates and a unified multi-task training strategy, EVF-SAM significantly outperforms prior SAM methods on RES datasets (e.g., RefCOCO/+/g) with only 1.32 billion parameters, achieving state-of-the-art results while reducing model size by nearly 82%. This method demonstrates the advantages of early multimodal fusion for referring expression segmentation.
weaknesses: 1.	The authors suggest that using a Multimodal Encoder with early fusion enhances SAM’s text-prompting capability. However, the study validates only two multimodal models—CLIP and BEIT-3. The efficacy of multimodal input in these cases could stem from the models’ unique training strategies, such as BEIT-3’s use of shared multi-head self-attention. It remains unproven whether other multimodal models, such as VLMO [1] and CoCa [2], would also benefit from this early fusion and Multimodal Encoder strategy. Further evaluation is needed to determine if these approaches can enhance SAM’s text capability across a broader range of multimodal models.
2.	The authors state that “Despite their advantages in terms of fewer parameters and faster inference speeds, these methods either achieve less competitive results or require vast amounts of data due to their lack of integration with foundation models.” However, in Table 2, there is no clear indication of the parameter counts or training data volumes for the comparison methods, making it difficult to assess the efficacy of the proposed approach. To strengthen the results, the authors should explicitly report parameter counts, training data volumes, and inference speeds for the comparative methods.
3.	The authors validate their method solely on the referring expression segmentation benchmark. With the introduction of text capability in SAM, there is a risk that EVF-SAM might overfit this specific benchmark, potentially reducing its effectiveness on unseen or less structured data. To demonstrate the robustness of EVF-SAM’s text-prompting ability, additional validation across multiple tasks, such as zero-shot semantic segmentation, zero-shot instance segmentation, and open-vocabulary segmentation, would be added.
4.	The paper primarily compares EVF-SAM against methods specific to the referring expression segmentation task. To provide a comprehensive comparison, the authors should also consider benchmarking against other methods aimed at enhancing SAM’s text capability, such as Open-Vocabulary SAM [3], SAM-CLIP [4], and AlignSAM [5].
5.	In the related work section, the authors should include a broader summary of research focused on SAM’s limitations and methods for enhancing its various capabilities. Relevant works include [6], [7], [8], and [9].

[1]Bao, Hangbo, et al. "Vlmo: Unified vision-language pre-training with mixture-of-modality-experts." Advances in Neural Information Processing Systems 35 (2022): 32897-32912.
[2]Yu, Jiahui, et al. "Coca: Contrastive captioners are image-text foundation models." arXiv preprint arXiv:2205.01917 (2022).
[3]Yuan, Haobo, et al. "Open-vocabulary sam: Segment and recognize twenty-thousand classes interactively." European Conference on Computer Vision. Springer, Cham, 2025.
[4]Wang, Haoxiang, et al. "Sam-clip: Merging vision foundation models towards semantic and spatial understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[5]Huang, Duojun, et al. "Alignsam: Aligning segment anything model to open context via reinforcement learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[6]Xie, Defeng, et al. "Edit everything: A text-guided generative system for images editing." arXiv preprint arXiv:2304.14006 (2023).
[7]Sun, Yanpeng, et al. "VRP-SAM: SAM with visual reference prompt." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[8]Yao, Jingfeng, et al. "Matte anything: Interactive natural image matting with segment anything model." Image and Vision Computing 147 (2024): 105067.
[9]Yu, Tao, et al. "Inpaint anything: Segment anything meets image inpainting." arXiv preprint arXiv:2304.06790 (2023).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: - Why do the authors use "zero-initialized sparse embeddings" to concatenate to multimodal embeddings before feeding into SAM's prompt encoder. Is there a specific reason?
- Why does the prompt encoder in SAM need to be trained or even included in this framework? Can we not simply input the multimodal embedding from the BEIT-3 directly into the mask decoder? What is the role of SAM's prompt encoder? Given the minor performance increase with finetuning SAM's prompts encoder (0.4 cIoU in Table 4) compared to freezing it, what is intention in finetuning it? 
- Why is using the image [CLS] token most effective in Table 5? 
- In Figure 5, the [CLS] token seems to attend to right upper image tokens (which are unrelated to the referring text nor salient parts of the image) in both the first and third attention maps. This may be explanable by this paper "Vision Transformers need Registers, Darcet et al., ICLR 2024" so checking it out may help. 
- Is there a possibility that BEIT-3 (Image+Text) does better than CLIP (Image+Text) because it has more computation in Table 1? If BEIT-3 (Image+Text) has less or similar computation, the conclusion that early fusion is better than late fusion would be more convincing.
soundness: 2
strengths: - Authors show that using a comparatively lightweight multimodal encoder can provide stronger performance than using heavier LLMs. 
- Authors show that investing in building an effective multimodal encoder may be more effective than building a more effective segmentation model (larger BEIT-3 is better than using larger SAM illustrated in "4.4 Ablations: Effects of Different Foundation Models.").
summary: Authors provide two main contributions in effectively extending SAM's general segmentation capability to Referring Expression Segmentation (RES). Firstly, they empirically show that prompting SAM with a multimodal prompts and employing 'early fusion' of image and text embeddings to produce such prompts is better than using unimodal, text prompts and employing 'late fusion'. Secondly, they propose EVF-SAM, which employs BEIT-3 as a multimodal encoder that produces a multimodal prompt from a downscaled version of the input image and referring text and inputs the multimodal prompt into SAM's original prompt encoder. Training the multimodal encoder and finetuning SAM's prompt encoder and mask decoder, EVF-SAM acquires strong performance on major RES benchmarks, especially on the longer and more difficult RefCOCO+ and RefCOCOg datasets given additional segmentation datasets for training. Authors highlight that their EVF-SAM is not only better in performance but also computationally efficient compared to LLM-based models for the smaller architecture of the multimodal encoder compared to LLMs.
weaknesses: - Lack of novelty
  - The proposed framework seems to be an addition of BEIT-3 as an multimodal encoder to SAM. No new modules nor techniques, but simply adding an existing, well-known multimodal encoder to another existing, well-known segmentation model. 

- Omitted references
   - When explaining previous work that explore the text-prompted abilities of SAM (line 051~), authors explain each work but omits references to them. Thus, the reviewer cannot be sure of these findings they draw from these work are valid. 
   - Findings used to justify the method also lacks references: ie. these sentences do not have references "However, the uncontrolled
length of the answering query introduces instability during both training and inference. Forcing the model to conform to a specific answering template can lead to language drift."

- Unclear writing 
  - Authors should better explain what they mean by "matches" in "Fast-SAM (Zhao et al., 2023) matches the similarity of CLIP (Radford et al., 2021) features between the text and Region of Interest (RoI) of image." 
  - Authors simply list these short sentences without sufficient explanation in Section 2.2: "PerceptionGPT (Pi et al., 2023) proposes an end-to-end architecture. u-LLaVA (Xu et al., 2023) supports multi-task. PSALM (Zhang et al., 2024) imports mask tokens to LMM input for better performance."
  - Authors claim that traditional RES models are limited since they "They fail to integrate with large-scale foundation models... e.g., SAM, LLaVA," but methods that employ LLMs and LMMs have "heavy computation burden", which seem contradictory. I understand that authors wanted to explain that traditional RES models are not using foundation segmentation/vision models and thus missing out on the generalizablity of these models and that models that use LLMs are inherently computationally heavy, but using LLaVA (which is LLM-based) as an example for the first point as a model that traditional models *should* try to incorporate and then claiming LLM-based models are heavy is inconsistent.
  - Authors explain that they use a special [semantic] token to alleviate semantic conflict but do not explain what the conflict is. Although I acknowledge the reference to UniLSeg, I think a short explanation would be very helpful.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses section.
soundness: 3
strengths: 1. The findings of early-fusion multimodal encoder models are intriguing, which is helpful for the community when designing text-prompted segmentation models. 

2. The experiments are comprehensive, with detailed empirical evaluations of the design choices of each part in the proposed method. The improvements are significant.
summary: This paper continues the research direction on text-prompted segmentation foundation models and proposes EVF-SAM with an early fusion paradigm. The authors find that multimodal prompts and early-fusion VLMs are beneficial for referring expression segmentation. Extensive experiments show the superiority of the proposed EVF-SAM model with much fewer parameters.
weaknesses: 1. The baseline of LLaVA is not fair enough since LISA mainly focuses on its Reasoning Segmentation ability. So it is necessary to fine-tune LLM-based models like LISA, GSVA with referring expression segmentation tasks to achieve a similar level. 

2. The training samples of LISA-7B models are 2 img/gpu $\times$ 8 gpu $\times$ 10 grad-accumulation $\times$ 500 steps/epoch $\times$ 10 epoch = 800k, however EVF-SAM has 128 samples/iter $\times$ 15k steps = 1,920k samples. The number of training samples should also be ablated carefully.

3. The novelty is somehow limited since the architecture and training methods of incorporating VLMs into RES, REC, GRES tasks have been investigated by previous works such as LISA, GSVA, PixelLM, GLaMM, u-LLaVA, PSALM. Using another early-fusion multimodal encoder seems not have enough innovation in terms of methodology. However, considering the abundant experiment results, it is not appropriate to deny the contribution of this paper. The authors are encouraged to provide more theoretical analyses and insights to justify why the early-fusion paradigm encoders bring such significant improvements.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Is the early fusion mechanism also applicable to other types of architectures, such as the decoder-only LLM-based architectures like PSALM and LISA?
2. It will be greatly beneficial to the community if the author can provide code and data to reproduce the experiments in this paper.
soundness: 2
strengths: 1. The proposed method achieves state-of-the-art performance in referring segmentation.
2. The paper is well-written and easy to read.
summary: This paper introduces EVF-SAM, which enhances the text-prompted segmentation ability of SAM by (1) leveraging a stronger multimodal encoder and (2) implementing early vision-language fusion. Through these techniques, it achieves new state-of-the-art performance on referring image segmentation tasks.
weaknesses: 1. Novelty:
> Using a stronger backbone (BEIT vs. CLIP) and early fusion are both expected to yield better performance, and it's unclear what additional insights this paper provides. Early or late fusions of multimodal inputs have been explored in many earlier referring segmentation works, such as LAVT[1], CoupAlign[2], and MaskGrounding[3], which all reach similar conclusions as this paper regarding the superiority of early fusion. The authors should clarify the unique insights gained from this paper.

2. Clarity:
> 1. Given that early fusion is presented as a core contribution, a detailed explanation of the process is expected. However, this information seems to be missing from the paper. The only details provided are a high-level diagram in Fig. 3 and unclear wording in line 230, which states, "We leverage the 'early-fusion' vision-language model as the Multimodal Encoder to generate prompt embeddings for SAM." It's unclear whether early fusion is a proposed novelty or part of the original BEIT-3 architecture. It appears to be the latter, suggesting that the main change is swapping CLIP with BEIT-3. Is that true?
> 2. In lines 237-240, the authors claim that prompt embeddings with special tokens lead to "instability during both training and inference." However, it seems that EVF-SAM also uses a special token (the [CLS] token after projections) as prompt embeddings, and it's unclear how this differs and resolves the previously mentioned problems.
> 3. For Tab. 7, providing mIoU results for ADE20K would be beneficial, as cIoU is not commonly used for this dataset, making comparisons with existing methods difficult.

3. Unfair comparisons:
> 1. The authors claim that EVF-SAM is superior to decoder-only Large Language Models like LAVT and PSALM based solely on referring segmentation performance. However, these LLM-based models are capable of more than just referring segmentation, including reasoning segmentation, generic segmentation, VQA, etc. This comparison is unfair unless the authors can demonstrate superiority in these other tasks as well.

4. Insufficient ablations:
> 1. Based on Tab. 3, early fusion experiments using the widely adopted CLIP model are lacking, and the improvements seen with early fusion in ViLT are modest (average 66.6 vs. 51.7). In contrast, for BEIT-3-Large, the improvement is significant, with performance increasing from an average of 52.3 to 78.0. This suggests that the early fusion mechanism may be particularly effective for BEIT-3-Large but may not be universally beneficial across other vision-language models. Given this, can the strong performance in Tab. 2 still be achieved with the more commonly used CLIP model?
> 2. The experiments in Tab. 2 are done using different training data, which makes fair comparison difficult. It will be great if the author can provide fairer comparisons using same training data as other works such as PSALM and LISA.

References:

[1] LAVT: Language-Aware Vision Transformer for Referring Image Segmentation, CVPR 2022

[2] CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation, NeurIPS 2022

[3] Mask Grounding for Referring Image Segmentation, CVPR 2024