Text Augmented Spatial Aware Zero-shot Referring Image Segmentation
Authors: Yucheng Suo
Linchao Zhu
Yi Yang
Venue: EMNLP 2023 Findings (2023)
Keywords: Zero-shot Referring Image Segmentation
Multi-modal Learning
Visual-text Matching
TL;DR: We study zero-shot referring segmentaion, using augmented text to rectify CLIP predictions.

Abstract: In this paper, we study a challenging task of zero-shot referring image segmentation. This task aims to identify the instance mask that is most related to a referring expression \textbf{without} training on pixel-level annotations. Previous research takes advantage of pre-trained cross-modal models, e.g., CLIP, to align instance-level masks with referring expressions. Yet, CLIP only considers the global-level alignment of image-text pairs, neglecting fine-grained matching between the referring sentence and local image regions. To address this challenge, we introduce a Text Augmented Spatial-aware (TAS) zero-shot referring image segmentation framework that is training-free and robust to various visual encoders. TAS incorporates a mask proposal network for instance-level mask extraction, a text-augmented visual-text matching score for mining the image-text correlation, and a spatial rectifier for mask post-processing. Notably, the text-augmented visual-text matching score leverages a $P$-score and an $N$-score in addition to the typical visual-text matching score. The $P$-score is utilized to close the visual-text domain gap through a surrogate captioning model, where the score is computed between the surrogate model-generated texts and the referring expression. The $N$-score considers the fine-grained alignment of region-text pairs via negative phrase mining, encouraging the masked image to be repelled from the mined distracting phrases. Extensive experiments are conducted on various datasets, including RefCOCO, RefCOCO+, and RefCOCOg. The proposed method clearly outperforms state-of-the-art zero-shot referring image segmentation methods.

Text Augmented Spatial-aware Zero-shot Referring Image Segmentation
Yucheng Suo
Linchao Zhu
Yi Yang
Zhejiang University, Hangzhou, China (suoych, zhul inchao, ywageiiccnanniiccccccco Jezzz...sseee......... edu. cn
CLIP-Only
TAS(Ours)
Abstract
this paper, we study a challenging task of zero-shot referring image segmentation. This task aims to identify the instance mask that is most related to a referring expression without training pixel-level annotations. Previous research takes advantage of pre-trained cross- modal models, e.g., CLIP, to align instance- level masks with referring expressions. Yet, CLIP only considers the global-level alignment of image-text pairs, neglecting fine-grained matching between the referring sentence and local image regions. To address this challenge, we introduce Text Augmented Spatial-awart (TAS) zero-shot referring image segmentation framework that training-frie and robust to various visual encoders. TAS incorporates mask proposal network for instance-leve mask extraction, text-augmented visual-exx match- ing score for mining the image-text correlation, and a spatial rectifier for mask post pott-pocessing Notably, the text-augmented visual-text match- ing score leverages a P-score and an N-score addition
SAM
SAM
Caption: women hoiltng
Caption:
imagge
woman right under the umbreta
Woman the right umbrata
CLIP
Negaliva taxt: man, umbretla...
Spatial Rectifier
X
Figure I: Main idea of TAS. CLIP solely calculates the cosine similarity between masked image embed- ding the text embedding, resulting in picking the wrong mask. TAS uses extra caption embedding, negative text embedding, and a spatial rectifier to rectify the CLIIP prediction, thereby picking the correct mask.
2021; Noh et al., 2015; Long et al., 2015), referring expression segmentation is a challenging task that requires identifying specific object described by referring expression (Yu et al., 2016; Yang et al., 2022; Wang et al., 2022d; Kim et al., 2022). The task has wide application scenarios such robot interaction, and image editing (Xu et al., 2023c). The acquisition precise referring expressions and dense mask annotations is labor-intensive, thereby limiting the practicality in real-world applications. Moreover, the quality and precision of the obtained annotations cannot be guaranteed regarding the annotation process. Therefore, we investigate zero-shot referring image segmentation to reduce labor costs training on annotations is not required under this setting.
typical visual-text matching score. The P-score is utilized close the visual-text domain gap through surrogate cap- tioning model, where the score is computed between the surrogate model-generated texts and the referring expression. The N -score con- siders the fne-graaine alignment of region-text pairs via negative phrase mining, encouraging the masked image to be repelled from mined distracting phrases. Extensive experiments conducted on various datasets, including Re- ICOCO, RefCOCO+, and RefCOCOg. The proposed method elearly outperforms state-of- the-art zero-shot referring image segmentation methods.
1 Introduction
Recently, zero-shot referring image segmenta- tion framework is proposed (Yu al., 2023). This framework initially extracts instance masks through an uff-the-shelf mask proposal network. Subse- quently, the appropriate mask selected by com-
Different from the traditional semantic segmen- tation tasks that predict masks belonging to pre- defined categories (Bolya et al., 2019; Strudel et al.,
#Corresponding author.

---

puting global-local CLIP (Radford al., 2021) similarity between the referring expressions and the masked images. However, the method focuses on the single object in each mask proposal and does not consider other distracting objects within the im- age. Moreover, since CLIP is trained on image-text pairs, directly applying it to the referring expres- sion segmentation task that requires fine-grained region-text matching could degenerate the match- ing accuracy (Zhong al., 2022). Another chal- lenge arises from the domain gap between masked images and natural images (Ding et al., 2022; Xu et al., 2022; Liang et al., 2023), which affects the alignment between masked images and referring expressions.
address this issue, we propose a spatial rectifier a post-processing module. For instance, to find out the mask corresponding to the referring expression "man to the left", we calculate the center point coordinates of all the masks and pick the mask with the highest -uxgeeegeeteed visual-text matching score from the left half of the masks.
Without modifying the CLIP architecture fur- ther fine-tuning our method facilitates CLIP pre- diction using a text-augmentigg manner, boosting the zero-shot referring expression segmentation performance. We conduct experiments and abla- tion studies on RefCOCO, RefCOCO+, and Re- iCOCOg. The proposed framework outperforms previous methods.
To this end, introduce Text Augmented Spatial-aware (TAS) zero-shot referring expres- sion image segmentation framework composed of a mask proposal network, a text-augmented visual- text matching score, and a spatial rectifier. We utilize the off-the-shell Segment Anything Model (SAM) (Kirillov et al., 2023) as the mask pro- posal network to obtain high-qualit, instance-level masks.
Related Work
2.1 Zero-shot Segmentation
As the rising of image-text pair pre-training multi- modal models like CLIP (Radford et al.. 2021), ALIGN (Jia et al., 2021), ALBEF (Li et al., 2021), researchers spend effort in combining cross-modal knowledge (Yang al., 2021) with dense predic- tion tasks like detection (Du et al.; Gu et al., 2021; Lin al., 2022; Bravo al., 2023; Wang et al., 2023a) and segmentation (Kim et al., 2023; Liang et al., 2022; Luo et al., 2022; Rao et al., 2021; Xu et al., 2023d; Zhou et al., 2021; Qin et al., 2023; Liang et al., 2023; Xu et al., 2023a,b). However, the text used in these works is restricted to object class words attributes (Li et al., 2022a). Re- cently, trend of unified segmentation networks brings dense prediction tasks to a new era (Wang et al., 2023c; Zou et al., 2023). A representative work is Segment Anything Model (SAM) (Kirillov et al., 2023). SAM takes any form of prompt (point, bounding box) to generate masks for a specific area, or to generate masks for all instances without any prompt. A series of works based on SAM aims to apply it in different using scenarios (Ji et al., 2023; He al., 2023; Li et al., 2023d; Cheng al., 2023).
To enhance the region-text aligning ability of CLIP and bridge the domain gap between the masked images and the natural images, a text- augmented visual-text matching score consisting of three components is calculated. The first score, called V -ccooe, is the masked image text matching score used for measuring the similarity between masked images and referring expressions. The sec- ond component is the P-score. bridges the text- visual domain gap by translating masked images into texts. Specifically, a caption is generated for each masked image, followed by calculating its similarity with the referring expression. The inclu- sion of captions enhances the consistency between the referring expressions and the masked images. To improve fine-grained region-text matching ac- curacy, we further repel distracting objects in the image by calculating the N-score. The N-score is the cosine similarity between masked images and negative expressions. We mine these negative expressions by extracting noun phrases from the captions of the input images. The mask that is most related to the referring expression is selected ac- cording to a linear combination of the above scores.
2.2 Referring Image Segmentation
Referring image segmentation differs from tradi- tional semantic segmentation and instance segmen- tation since needs comprehension for a sentence describing specific object (Hu et al., 2016; Yu et al., 2016; Subramanian al., 2022). Plenty of fully supervised methods achieve impressive per- formance (Wang et al., 2022d; Yang et al., 2022;
Another challenge arises from the limitation of CLIP in comprehending orientation descriptions, as highlighted by (Subramanian et al., 2022). To

---

Caption Embadding
Cantrantive Augmented Score
Mask Generation
M&B
Captioner
CLIP Text Encoder
S
Referring Expression Embedding
d
Input Image
SAM
Input Retering expression:
girl sitting the slumped the bed
CLIP
+
Spatial Rectifier
Text
S
Masked Image Embedding
Output Mask
CLIP Encoder
n
M&B
Mask and Bluming
Negative Text Embadding
gri sitingg to the left slumped the bed
Negative Text Miner
a man bed
CLIP Encoder
M&C
Mask and Crapping
Captioner
> man sitiing on bed
Figure 2: Overall pipeline of TAS. Given an input image, we first leverage SAM to obtain instance- masks. Each mask proposal is applied to the original image calculate the V-score with the referring expression. The captioner generates a caption for each masked image to calculate the P-score. Negative expressions are mined to calculate N -cocoe. The mask with highest score picked after post-processing of a spatial rectifier.
Xu et al., 2023c: Liu et al., 2023; Kim et al., 2022: Luo et al., 2020; Ding et al., 2021; Huang et al., 2020), yet these works require pixel-level anno- tations along with precise referring expressions which are labor-intensive. Recently, weakly su- pervised method is proposed, which trains a net- work only based the image text pair data (Strudel al.. 2022). Another work goes step further by utilizing CLIP directly retrieve FreeSOLO (Wang et al., 2022c) proposed masks without any training procedure (Yu et al., 2023).
bridge the gap between masked images and natural images (Liang et al., 2023) without modifying the model architecture. To achieve the goal, our intu- ition is to exploit fine-gra grained regional information using positive and negative texts since text descrip- tions summarize the key information in masked images. Therefore, we propose new Text Aug- mented Spatial-aware (TAS) framework consisting of three main components: mask proposal net- work, text-augmented visual-texi matching score, and a spatial rectifier. The mask proposal network first extracts instance-level mask proposals, then the text-augmented visual-text matching score is calculated between all masked images and the refer- ring expression to measure the similarity between masks and the text. After post-processed by the spa- tial rectifier, the mask most related the referring expression selected.
2.3 Image Captioning
Image captioning, classic multi-modal task, aims generate a piece of text for an image(Li al., 2023b; Changpinyo et al., 2021; Mokady et al., 2021; Wang ct al., 2022a, 2023b, 2022a). As the training data amount gets tremendous, the parame- ter of the state-of-the-art models grows rapidly (Li et al., 2022c; Zhang et al., 2022; Bao et al., 2022; Chung al., 2022; Wang et al., 2022b; Alayrac al., 2022). Recent advance in large language models enriches the text diversity of the gener- ated captions (Zhu al., 2023; Brown al., 2020; Ouyang et al., 2022). this paper, we adopt the widely used image captioning network BLIP-2 (Li et al., 2023a).
3.2 Mask Proposal Network
Previous works (Yu al., 2023; Zhou al., 2021) indicate that is suboptimal directly apply CLIP on dense-ping prediccte tasks. Therefore, we follow previous works (Yu et al., 2023; Liang et al., 2023; Xu et al., 2022) to decompose the task into two procedures: mask proposal extraction and masked image-text matching. To obtain mask proposals, we adopt the strong off-the-shee mask extractor, i.c., SAM (Kirillov al., 2023), the mask pro- posal network. The mask proposal network plays vital role as the upper bound performance heavily relies on the quality of the extracted masks.
3 Method
3.1
Overall Framework
This paper focuses on zero-shot referring expres- sion image segmentation. Our main objective to enhance the fine- aoee-graned region-text matching capability of image-text contrastive models and
FreeSOLO vs. SAM. Zero-shot referring expres- sion segmentation is to identify specific object

---

Refeniing expression: girl
referring expression T, SAM extracts series of binary masks MI from the image. Every mask pro- posal m E M is applied to the input image I. Then the foreground area of the masked image is cropped and fed to the CLIP visual encoder following the approach of previous works (Yu et al., 2023; Liang et al., 2023). The visual feature and text feature extracted by CLIP are used to calculate the cosine similarity. This procedure can be formulated as:
Image
FreeSOLO
SAM
Mask
Referring expression: player in red and white
Image
FreeSOLO
SAM
GT Mask
Figure 3: Comparison between mask proposal net- works. Mask that has the largest IoU with the ground truth mask is visualized.
Im crop m): Sy (om (T,))
(2)
according to the referring expression. Hence, mask proposal networks with stronger ability to dis- tinguish instances can yield higher upper-bound performance. Previous work leverage FreeSOLO (Wang et al., 2022c), a class-agnostic instance seg- mentation network, to obtain all masks. However, we empirically find that the recently proposed SAM (Kirillov et al., 2023) shows strong performance in segmenting single objects. Figure 3 presents qualitative comparison between mask proposal networks. SAM exhibits superior performance in separating objects. achieving higher upper-bound performance. We observe that FreeSOLO faces challenges in distinguishing instances under occlu- sion or in clustered scenarios, whereas SAM is capable of handling such situations effectively. To achieve higher performance, we opt for SAM as the mask proposal network.
where crop represents the masking and cropping operation. Eg and Eg indicate the CLIP visual en- coder and the CLIP text encoder, respectively. cos means the cosine similarity between two types of features. We term the result as S", which represents the visual-text matching score.
Note that the CLIP vision encoder and the CLIP text encoder can be substituted by any image-text contrastive pre-trained models.
P-score. As mentioned earlier, the domain gap between the natural images and masked images af- fects the visual-text alignment. To bridge this gap, we introduce P-score to improve the alignment quality by leveraging surrogate captioning model. The idea to transfer the masked images into texts, which provides CLIP with complementary object information. Specifically, we use an image caption- ing model to generate a complementary caption C for each masked image. We encode the captions using the CLIP text encoder and calculate the co- sine similarity with the referring expressions. The procedure can be summarized as:
3.3 Text-augmented visual-text matching score
The mask proposal network provides instance-level masks, but these masks do not inherently contain semantics. To find the mask most related to the referring expression, the typical method is to calcu- late the cosine similarity between masked images and the referring expression using image-text con- trastive pre-trained models like CLIP. One of the is- sues is that CLIP may be incapable of fine-grained region-text matching (Zhong al., 2022) since it trained on image-text pairs. Moreover, the domain gap between masked images and natural images degenerates the masked image-text match- ing accuracy. To alleviate these issues and facilitate CLIP prediction, we mine regional information us- ing complementary texts. Therefore, we introduce text-augmented visual-text matching score com- posed of a V V-score, P-score, and an -sccore. V -ccore. Given an input image I E IR HW x3 and
SP cos (B.)
(3)
SP is the P-score measuring the similarity between captions and referring expressions. Note that the p- score is flexible to any captioning model. However, the effectiveness of SP highly depends on the qual- ity of generated captions. Better caption models could bring higher performance.
-sccore V-score and P-score promote alignment between masked images and referring expressions. Considering the existence of many objects in the image being unrelated to the referring expression, further propose an N-score to filter out these objects. To identify distracting objects, we collect negative expressions for these objects. Then we regard the similarity between masked images and negative expressions as a negative N-score. The

---

effectiveness of the score depends on these nega- tive expressions. To mine unrelated expressions, we first generate an overall caption for the input im- age. The overall caption summarizes all objects in the image. We then extract noun phrases from the caption using spacy (Honnibal and Johnson, 2015) and regard them as potential negative expressions. Note that there might be phrases indicating the same object in the referring expression. To avoid this situation, we use Wordnet (Miller, 1995) to eliminate the phrases that contain synonyms with the subject in the referring expression. Specifically, we calculate the path similarity of the two synsets to determine whether to eliminate the synonyms. Empirically, we find the strict rules help TAS to identify distinct objects in these datasets. For in- stance, we believe "young man" and "the boy" are not synonyms. This ensures that the negative ob- jects identified are distinct from the object men- tioned in the referring expression. The remaining noun phrases set Tn is used to calculate the cosine similarity with the masked images. S is defined as the averaged similarity value over the phrases:
3.4 Spatial Rectifier
As revealed in (Subramanian al., 2022), the text- image pair training scheme does not consider spa- tial relations. In other words, CLIP cannot distin- guish orientation descriptions such as "the left cat" or "the giraffe to the right". To this end, we propose spatial resolver for post-processing forcing the framework to select masks from the specific region. The procedure can be decomposed into three steps: orientation description identifica- tion, position caleulation, and spatial rectifying. Orientation description identification. First, we extract descriptive words for the subject of the refer- ring expression T, vía spacy (Honnibal and John- son, 2015) and check whether there are orientation words like "up, bottom, left, right". If no orienta- tion words are found in the descriptive words, we do not apply spatial rectification.
a
Position calculation. Second, to spatially rectify the predictions, need the location information of each mask proposal. The center point of each mask used as a proxy for location. Specifically, the center point location of each mask is calculated by averaging the coordinates of all foreground pix- els.
1
S m
Σ (())
(4)
I
TET,
Spatial rectifying. After obtaining center point locations, we choose the mask with the highest overall score S under the corresponding area of the orientation. For instance, we pick the mask for the expression "the left cat" from the masks whose center point location is in the left half of all the cen- ter points. Having this post-procsssing procedure, we restrict CLIP to pay attention to specific areas when dealing with orientation descriptions, thereby rectifying wrong predictions.
It is worth mentioning that S" is negative score since it measures the probability of masked im- age representing an object unrelated the target referring expression. We enhance fine-grained ob- ject region-text matching by eliminating regions for distracting objects. S" is also flexible to caption- ing models, while detailed captions help to capture more negative expressions.
The text-augmented visual-text matching score. The final text-augmented visual-text matching score be obtained by linearly combining the three above-mentioned scores since all scores the cosine similarity calculated in the common CLIP feature space. The output mask is the one with the highest score.
Experiments
4.1 Dataset and Metrics
The proposed method is evaluated on the widely used referring image segmentation datasets, i.e. Re- fCOCO, RefCOCO+, and RefCOCOg. All images in the three datasets come from the MSCOCO dataset and are labeled with carefully designed referring expressions for instances. We also re- port the performance on the PhraseCut test set. In terms of the metrics, we adopt overall Intersection over Union (oloU), mean Intersection over Union (mloU) following previous works.
S,n SY + SP + AS
(5)
m
m argmax S.. MCM
(6)
The final mask n is selected by choosing the one with the highest S. Without changing the fea- ture space and modifying the structure, the text- augmented visual-text matching score enhances fin- in-grained region-text matching only using aug- mented texts.

---

Visual Encoder
RefCOCO
RefCOCO+
RefCOCOg
Metric
Method
Val
TestA
TestB
Val
TestA
TestB
Val(U)
Tes((U)
Test(G)
Text-only
18.65
17.60
18.32
21.78
21.68
20.58
25.11
25.48
25.93
CLIP Surgery
18.04
14.74
21.28
18.39
14.34
22.98
20.44
21.80
21.23
Score Map
20.18
20.52
21.30
22.06
22.43
24.61
23.05
23.41
23.69
GradCAM
23.44
23.91
21.60
26.67
27.20
24.84
23.00
23.91
23.57
oloU
ResNet-50
Global-Local
24.58
23.38
24.35
25.87
24.61
25.61
30.07
29.83
29.45
Global-Local
20.54
22.15
20.35
24.41
25.73
24.58
26.75
28.86
29.01
CLIP-only
23,41
24.77
24.07
29.83
35,46
26.25
31.26
32.47
33.08
TAS (Ours)
29.75
32.85
28.23
33.58
40.65
27.78
36.80
37.06
38.03
Region Token
11,56
12.37
11.35
12.54
14.07
12.22
10.87
11,51
11.74
Global-Local
24.88
23.61
24.66
26.16
24.90
25.83
31.11
30.96
30.69
VIT-B/32
Global-Local
22.43
24.66
21.27
26.35
30.80
22.65
27.57
27.87
27.80
CLIP-only
23.01
23.18
24.27
30.05
33.74
27.01
30.66
31.41
32.27
TAS (Ours)
29.53
30.26
28.24
33.21
38.77
28.01
35.84
36.16
36.36
VIT-S/16
TSEG
25.95
22.62
23.41
Text-only
26.17
24.19
25.98
30.00
29.66
29.61
35.83
35.74
36.21
CLIP Surgery
25.03
22.71
27.12
26.01
22.20
30.18
29.26
30.07
29.43
Score Map
25.62
26.66
25.17
27.49
28.49
30.47
30.13
30.15
31.10
GradCAM
30.22
31,90
27.17
33.96
25.66
32,29
33.05
32,50
33.25
mloU
ResNet-50
Global-Local
26.70
24.99
26.48
28.22
26.54
27.86
33.02
33.12
32.79
Global-Local
32.73
35.31
30.09
37.74
40.69
34.93
41.62
42.88
43.96
CLIP-only
34.16
35.80
31.70
41.07
46.61
34.50
43.73
44.09
45.08
TAS (Ours)
39.91
42.85
35.85
43.99
50.58
36.44
47.68
47.41
48.69
Region Token
17.06
18.02
16.28
18.83
20.31
17.78
16.33
16.88
17.31
Global-Local
26.20
24.94
26.56
27.80
25.64
27.84
33.52
33.67
33.61
VIT-B/32
Global-Local
32.93
34,93
30.09
38.37
42.05
32.65
42.02
42.02
42.67
CI.IP-only
33.01
33.60
31.46
40.59
43.73
35.25
42.66
42.59
44.37
TAS (Ours)
39.84
41.08
36.24
43.63
49.13
36.54
46.62
46.80
48.05
Table Performance on RefCOCOg, RefCOCO+ and RefCOCO datasets. and denote the UMD and Google partition of the RefCOCOg datasct respectively. TAS outperforms all baseline methods in terms both oloU and mloU on different CLII visual backbones. Note that methods use mask proposals provided by SAM. f indicates using FreeSOLO to extract masks.
4.2 Implementation Details
4.3 Baselines
Baseline methods can be summarized into two types: activation map-based and similarity-based For activation map-based methods, we apply the mask proposals to the activation map, then choose the mask with the largest average activation score. Following previous work, Grad-CAM (Selvaraju et al.), Score Map (Zhou et al., 2021), and Clip- Surgery (Li et al., 2023c) are adopted. Note that Score Map is acquired by MaskCLIP. Similarity- based methods are to calculate masked image-text similarities. Following previous work, we adopt Region Token (Li et al., 2022b) which utilizes mask proposals to filter the region tokens in every layer of the CLIP visual encoder, Global-Local (Yu et al., 2023) uses Freesolo as the mask proposal network and calculates the Global-Loca image and text sim- ilarity using CLIP. Note that for fair comparison, also report the results using SAM. Text-only (Li et al., 2023a) to calculate the cosine similarity between the captions for masked images and the referring expressions. This baseline to test the relevance of the caption and referring expression.
We adopt the default VIT-H SAM, the hyper- parameter "predicted iou threshold" and "stability score threshold" are set to 0.7, "points per side" is set to For BLIP-2, we adopt the smallest OPT- 2.7b model. As for CLIP, we use RN50 and ViT- B/32 models with an input size of 224 x 224. We set À to 0.7 for RefCOCO and RefCOCO+, 1 for RefCOCOg, and 0.1 for all datasets. When us- ing SAM the mask generator, we noticed a high number of redundant proposals. SAM often pro- duces masks for different parts of the same object, leading to issues with visual semantic granularity. In our approach, we have implemented a straight- forward yet effective method to filter out irrelevant mask proposals, thereby reducing semantic ambi- guity. Specifically, after generating all initial mask proposals, we measure the overlap between any masks and eliminate smaller masks that are subsets of larger ones. important to note that in the experiments section, methods are evaluated based on these refined mask proposals.

---

Visual Encoder
CuraseCut
Modules
RefCOCO
Method
oloU
mloU
SP
S
Spatial
oloU
mloU
Global-Local
23.64
23.01
33.01
25.44
35.94
ViT-B/32
25.64
24.66
TAS(Ours)
27.11
37.03
ResNet-50
25.00
24.51
29.53
39.84
Table 2: oloU and mIoU results PhraseCut dataset.
Table 4: Ablation study the performance improve-
ment of each proposed module. We report oloU and mloU on the validation set of RefCOCO using VIT-B/32 CLIP visual backbone.
Refcoco
RefCOCO
oloU
mloU
oloU
mIoU
27.41
37.89
27.46
38.54
0.05
28.75
39.35
0.3
28.62
39.56
0.1
29.53
39.84
0.5
29.23
39.97
combination of different types of scores. To ex- plore whether the score is sensitive toward the weights a and X, we conduct an ablation study. Results are shown in Table 3, a and are tuned separately. TAS not sensitive to A, we select 0.7 to achieve a balance of mloU and oloU improve- ment. A large a harms the performance, therefore we set the value to 0.1.
0.2
29.17
39.13
0.7
29.53
39.84
26.93
36.39
0.9
28.96
39.18
Table 3: Ablation study on the impact of A and a. We separately change the value of and À and test the oloU and mloU on the RefCOCO validation set. ViT-B/32 CLIP visual backbone is adopted here.
CLIP-only (Radford al., 2021) is simple base- line that directly calculates the similarity between the cropped masked image and referring expres- sion. We also compare with TSEG (Strudel al., 2022), a weakly supervised training method.
Importance of the proposed modules. To fur- ther prove the effectiveness of the proposed text- augmented visual-text matching score and the spa- tial rectifier, we conduct an ablation study on the validation set of RefCOCO, The mloU and oloU results are reported with different combinations of the modules in Table 4. Scap and S are the P-score and the N-score respectively. "Spatial" represents the spatial rectifier aforementioned. The first line in the table is the result that only uses Simg, which is also the CLIP-Only baseline result. From the table, we observe that all modules contribute to performance improvement. In particular, the Spatial rectifier plays vital role in the ReiCOCO dataset since RefCOCO contains many orientation descriptions.
4.4 Results
Performance on different datasets. Results on RefCOco, RefCOCO+ and RefCOCOg are shown in Table 1. For a fair comparison, we reimplement the Global-Local (Yu et al., 2023) method using masks extracted from SAM. TAS outperforms all baseline methods in terms of oloU and mloU. Previ- works that leverage CLIP visual encoder activa- tion maps perform poorly in all datasets. Compared with the previous SOTA method using FreeSOLO to extract masks, TAS surpasses both metries, especially in mloU, We also report mloU and oloU results on the test set of the PhraseCut dataset in Table 2. Our method also outperforms the previous method.
Influence on the input format of masked im- ages. In table 5, we study two input types of masked images for the BLIP-2 and CLIP. The first method is cropping, which is widely used previ- ous works(Xu et al., 2022: Liang al., 2023; Yu et al., 2023). Another method blurring (Subra- manian et al., 2022), we blur the background of the cropped area using a Gaussian kernel. Blur- ring make the model recognize the mask area with background information. From the table, we find that for the captioning model BLIP-2, blurring is better than crop. However, cropping is better than blurring for CLIP. We suppose the reason the cropping left black background which helps CLIP to focus on the foreground object. However, for BLIP-2, blurring helps generate context-aware de-
Qualitative analysis. Figure 4 shows the quali- tative comparison of TAS and previous methods. Note that all the masks extracted by SAM. TAS is able to comprehend long complex referring ex- pressions and pick the most accurate object mask. With the help of the spatial rectifier, TAS deals well with orientetion-related referring expressions.
4.5 Ablation Study
Sensitive toward a and 3. We propose the text- augmented visual-text matching score, a linear

---

Referring Expression:
kid streaching or whatever he is doing
Image
Score map
GradCAM
CLIP-Only
Global-LLcal(SAMM
Ours
GT
Referring Expression: left man jumping to catch frisbee
Image
map
GradCAM
CLIP-Only
Global-Local (SAM)
Ours
Referring Expression: a wooden chair sitting in the corner next to a few rolls of toilet paper
Image
Score map
GradCAM
CLIP-Only
Global-Local (SAM)
Ours
GT
Figure 4: Qualitative results of different methods. All methods are compared using SAM mask proposals.
Mask Input Type
RefCOCO
RefCOCO+
RefCOCOg mloU
Method
BLIP-2
CLIP
oloU
mloU
oloU
mloU
oloU
crop
crop
28.87
39.08
CLIP
33.74
43.73
30,66
42.66
blur
blur
27.17
37.85
CLIP-TAS
38.77
49.13
35.84
46.62
crop
blur
27.07
37.43
BLIP-2-ITC
38.78
51.75
32.99
46.91
blur
crop
29.53
39.84
BLIP-2-TAS
43.48
54.90
37.97
50.07
Table 5: Ablation study the input type of masked images. oloU and mloU results are reported on the vali- dation split of RefCOCO using ViT-B/32 CLIP visual backbone.
Albef-ITC 7.22
11.70
10.55
15.35
Albef-TAS 24.59
35.10
26.58
37.20
Table 7: Ablation study on image text contrastive model. We report oloU and mIoU the testA split of Ref- COCO+ and RefCOCOg UMD validation split using Albef and BLIP-2 image-text contrastive model.
Visual Encoder
RefCOCO+
RefCOCOg mloU
Model
oloU
mloU
oloU
ResNet-50
35.46
46.61
31.26
43.73
VIT-B32
33.74
43.73
30.66
46.62
sults are shown in Table 6, we find that the cap- tioning model has little affection on performance. Better captioning models bring better mloU and 46.62
ResNet-50
40.65
50.58
36.80
47.68
BLIP-2
VIT-B32
38.77
49.13
35.84
ResNet-50
40.10
50.55
35.43
46.79
oloU performance.
GIT
VIT-B32
38.09
48.42
34.99
46.02
Is TAS generalizable to other image-text con- trastive models? To explore whether TAS is gen- eralizable to other image- contrastive models, conduct an ablation study, and the results are shown in Table 7. On BLIP-2 (Li al., 2023a) and ALBEF (Li et al., 2021), TAS makes impressive im- provements. We believe TAS is versatile method that helps any image-text contrastive model.
Table Results on GiT-base image captioning model. We report oloU and mloU on the testA split of Ref- COCO+ and RefCOCOg UMD validation split using kinds of CLIP visual backbone.
scriptions, enhancing comprehension of the refer- ring expression.
Importance of the image captioning model. Our intuition is to use texts to enhance region-text align- ment and bridge the domain gap between natural images and masked images. The quality of texts depends on the captioning model. To explore the importance of the captioning model, we substitute the BLIP-2 model with Guse captioning model (Wang et al., 2022a) and test the performance. Re-
Is TAS practicable in real-world scenarios? TAS does not require high computing resources. All experiments
conducted on single RTX 3090. which is applicable in real-world applications. The GPU memory consumption for the entire pipeline about 22GB, including mask generation (SAM), captioner (BLIP2), and masked image-text match- ing (CLIP). We also test the inference speed on a

---

random selection of 500 images on single RTX 3090. The CLIP-Only bascline method (mask gen- eration + masked image-text matching) obtains 1.88 seconds per image. The Global-Local method costs 2.01 seconds per image. Our method TAS (mask generation + captioner+ + masked image-text matching) achieves 3.63 seconds per image. By employing strategies like 8-bit blip-2 models and FastSAM (Zhao al., 2023), it would be possible enhance the efficiency under constrained compu- tational resources.
research topic directly identify the most appro- priate mask from noisy proposals. In other words, future works may work on designing a more robust method to deal with the semantic granularity of the mask proposals. Recent work uses diffusion mod- cls as a condition to work on this problem (Ni et al., 2023). Finally, the understanding of the metaphor and antonomasia within the referring expression remains insufficient. We observe there are expres- sions requiring human-level comprehension which is extremely hard for current image-text models. Future work may benefit from the comprehension and reasoning ability of Large Language Models (LLM).
5 Conclusion
this paper, we propose a Text Augmented Spatial- aware (TAS) framework for zero-shot referring im- age segmentation composed of a mask proposal network, a text-augmented visual-text matching score, and spatial rectifier. We leverage ohf-the- shell SAM to obtain instance-level masks. Then the text-augmented visual-text matching score is calculated to select the mask corresponding to the referring expression. The score uses positive text and negative text to bridge the visual-tex domain gap and enhance fine-grained region-text alignment with the help of caption model. Followed by the post-processing operation in the spatial rectifier, TAS is able to deal with long sentences with ori- entation descriptions. Experiments RefCOCO, RefCOCO+, and RefCOCOg demonstrate the ef- fectiveness of our method. Future work may need to enhance comprehension of hard expressions over non-salient instances the image. One potential way is to leverage the reasoning ability of Large language models like GPT4.
7
A Acknowledgement
This work is partially supported by Major pro- gram of the National Natural Science Foundation of China (Grant Number: T2293723). This work is also partially supported by the Fundamental Re- search Funds for the Central Universities (Grant Number: 226-2023-00126, 226-2022-00051).
8 Ethics Statement
The datasets used in this work are publicly avail- able.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems,
6 Limitations
Hangbo Bao, Wenhui Wang. Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subho- jit Som, Songhao Piao, and Furu Wei. 2022. Vlmo: Unified vision-language pre-training with mixture-of- modality-experts. Advances in Neural Information Processing Systems, 35:32897-32912.
While our approach yields favorable results across all datasets based on mloU and oloU metrics, there exist certain limitations that warrant further investi- gation. One such limitation is that SAM occasion- ally fails generate ideal mask proposals, thereby restricting the potential for optimal performance. Additionally, the effectiveness of our approach is contingent upon the image-text contrastive model employed. Specifically, we have found that the BLIP-2 image-text contrastive model outperforms CLIP, whereas the Albef image-text contrastive model shows poor performance when applied to masked images.
Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. 2019. Yolact: Real-tima instance segmentation. In Proceedings of the IEFF/CVF international con- ference on computer vision, pages 9157-9166.
M. Bravo, S. Mittal, S. Ging, and T. Brox. 2023. Open- vocabulary attribute detection. In IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR).
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot
Another potential limitation of TAS is the abil- ity to deal with complex scenarios. A potential

---

learners. Advances in neural information processing systems, 33:1877-1901.
IEEE/CVF conference on computer vision and pat- tern recognition, pages (0488-10499
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 3558-3568.
Wei Ji, Jingjing Li, Qi Bi, Wenbo Li, and Li Cheng- 2023. Segment anything is not always perfect: An investigation of sam different real-world applica- tions, arXiv preprint arXiv:2304.05750
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, HieuT, Pham, Quoc Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up vi- sual and vision-language representation learning with noisy text supervision.
Yangming Cheng, Liulei Li, Yuanyou Xu. Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang. 2023. Segment and track anything. arXiv preprint WxXiv2305.06558.
K Kwanyoung Kim, Yujin Oh, and JongChul Ye, 2023, Zegot: Zero-shot segmentation through optimal trans- port of text prompts.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetie finetumed language models. arXiv preprint arXiv:2210. 10.1 11416.
Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak. 2022. Restr: Convolution- free referring image segmentation using transform- ers. In Proceedings of the IEFF/CVE Conference Computer Vision and Pattern Recognition, pages 18145-18154.
Henghui Ding, Chang Liu, Suchen Wang. and Xudong Jiang. 2021. Vision-language transformer and query generation for referring segmentation. Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 16321-16330.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. arXiv preprint arXiv:2304.0266 02.444.....3....
Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. 2022. Decoupling pero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 11583- 11592.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranfil. 2022a. Language- driven semantic segmentation. arXiv preprint arXiv:2201.03546
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model.
Jiahao Li, Greg Shakhnarovich, and Raymond A Yeh. 2022b. Adapting clip for phrase localization without further training. arXiv preprint (xXix:2204.03444
Xiuye Gu, Tsung-Yo Yi Lin, Weicheng Kuo, and Yin Cui. 2021. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint 1.Xv:22044.3921.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. arXiv preprint arXiv:2301.1255
Sheng He, Rina Bao. Jingpeng Li, Ellen Grant, and Yangming Ou 2023. Accuracy of segment-anything model (sam) in medical image segmentation tasks. arXiv preprint WxXv:2304.0932.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022c. Blip: Bootstrapping language-image pre- mre-training for unified vision- understand- ing and generation. In International Conference on Machine Learning, pages
Matthew Honnibal and Mark Johnson. 2015. An im- proved non-monotonic transition system for depen- dency parsing. In Proceedings of the 2015 Confer- ence on Empirical Methods in Natural Language Processing, pages 1373-1378, Lisbon, Portugal. As- sociation for Computational Linguistics.
PMLR.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705.
Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. 2016. Segmentation from natural language expres- sions. In Computer Vision-ECCV 2016: 14th Eu- ropean Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 108-124. Springer.
Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. 2023b. Decap: Decoding clip latents for zero-shot captioning via text-only training. arXiv preprint arXiv:2303.03032.
Shaofei Huang, Tianrui Hui, Si Liu, Guanbin Li, Yun- chao Wei, Jizhong Han, Luoqi Liu, and Bo Li. 2020. Referring image segmentation via mooss-modal pro- gressive comprehension. In Proceedings of the
Yi L., Hualiang Wang, Yiqun Duan, and Xiaomeng Li. 2023c. Clip surgery for better explainability with en- hancement in open-vocabulary tasks, arXiv preprint wrXiv.2304.06653.

---

Yuheng Li, Mingzhe Hu, and Xiaofeng Yang. 2023d. Polyp-sam: Transfer sam for polyp segmentation. arXiv preprint 2.xiv22305.00200
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 13:27730-277744
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Va- jda, and Diana Marculescu. 2022. Open-vocabulary semantic segmentation with mask mask-ad clip.
Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi. Xuefeng Xiao, Yitong Wang, Rui Wang. Shilei Wen, Xin Pan, et al. 2023. Freeseg: Unified, universal and vecabbabbyy image segmentation. In Pro- ceedings of the IEFF/CVF Conference on Computer Vision and Patter Recognition, pages 19446-19455
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yi- nan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. 2023. Open-vocabulary se- mantic segmentation with mask-adapted clip. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061-7070.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence machine learning, pages 8748 8763, PMLR.
Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. 2022. Learning object-language alignments for open-vocabulary object detection.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yan- song Tang. Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. 2021. Denseclip: sanguage-guides dense prediction with context-aware prompting.
Jiang Liu, Hui Ding, Zhaowei Cai. Yuting Zhang. Ravi Kumar Satzoda, Vijay Mahadevan, and R Man- matha. 2023. Polyformer: Referring image segmen- tation as sequential polygon generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Puttern Recognition, pages 18653-18663.
Ramprasaath Selvaraju. Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization.
Jonathan Long, Evan Shelhamer, and Trevor Darrell, 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440.
Robin Strudel, Ricardo Garcia, Tvan Laptev, and Cordelia Schmid. 2021. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference computer vi- sion, pages 7262-7272.
Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. 2020. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Pro- ceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 10034-10043.
Robin Strudel, Ivan Laptev, and Cordelia Schmid. 2022. Weakly-superviser segmentation of referring expres- sions. arXiv preprint vrXiv:2205.04777
Sanjay Subramanian, Will Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. 2022. Reclip: A strong zero-shot baseline for re- ferring expression comprehension. arXiv preprint arXiv:220..0599).
Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. 2022. Segelip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation.
Jianfeng Wang, Zhengyuan Yang. Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022a. Git: A generative image-to- textet transformer for vision and language. arXiv preprint arXiv:2205. 20..222550.00000 14100.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(1 (8(11)) 41.
Ron Mokady, Amir Hertz, and Amit H Bermano. 2021. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.0973:
Luting Wang, Liu, Penghui Du, Zihan Ding. Yue Liao, Qiaosong Qi. Biaolong Chen, and Si Liu. 2023a. Object-aware distillation pyramid for open- vocabulary object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- lern Recognition, pages 11186-11196..
Minheng Ni, Yabo Zhang, Kailai Feng, Xiaoming Li,
Yiwen Guo, and Wangmeng Zuo. 2023. Ref-diff:: Zero-shot referring image segmentation with genera- tive models. arXiv preprint arXiv:2308. 16777.
Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2022b. Image as foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint 2200 10442.
Hyeonwoo Noh. Seunghoon Hong, and Bohyung Han. 2015. Learning deconvolution network for semantic segmentation. In Proceedings of the IEEE interna- tional conference on computer vision, pages 1520- 1528.

---

Wenhui Wang, Hangbo Bao, Li Dong. Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, al. 2023b. Image foreign language: Beit pretraining for vision and vision-lani laaguagg tasks. In Pruceedings of the IEEF/CVE Conference on Com- puter Vision and Pattern Recognition, pages 19175- 19186.
Zhao Yang, Jiagi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. 2022. Lavt: Language-aware vision transformer for referring im- segennnee
age segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 18155-18165.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. 2016. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings. Part II 14, pages 69-85. Springer.
Xinlong Wang. Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. 2022c. Frecsolo: Learning to seg- ment objects without amnotations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14176-
Seonghoon Yu, Paul Hongsuck Seo, and Jeany Son. 2023. Zero-shot referring image segmentation with global-local context features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 19456-19465.
Xinlong Wang. Xiaosong Zhang, Yue Cao, Wen Wang. Chunhua Shen, and Tiejun Huang. 2023c. Seggpt: Segmenting everything context. arXiv preprint arXiv:2304.03284.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprial arXiv:2205.0000
Zhaoqing Wang, Yu Lu, Qiang Li, Xungiang Tao, Yan- dong Guo, Mingming Gong, and Tongliang Liu. 2022d. Cris: Clip-driven referring image segmenta- tion. In Proceedings of the IEFT/CVF conference on computer vision and pattern recognition, pages 11686-11695 11695.
Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. 2023. Fast segment anything. arXiv preprint M.Xi:2330612155
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. 2023a. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 2955-2966.
Yiwu Zhong, Jianwei Yang. Pengehuan Zhang, Chun- yuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. 2022. Regionclip: Region-based language-image pretrain- ing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16793-16803.
Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. 2023b. Learning open- vocabulary semantic segmentation models from nat- ural language supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pat- tern Recognition, pages 2935-2944.
Chong Zhou, ChenChange Loy, and Bo Dai. 2021. Ex- tract free dense labels from clip.
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaogian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. 2023. Chatgpt asks, blip-2 answers: Automatic ques- tioning towards enriched visual descriptions. arXiv preprint 2436042303..6554.
Li Xu, Mark He Huang, Xindi Shang, Zehuan Yuan, Ying Sun, and Jun Liu. 2023c. Meta compositional referring expression segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14478-19487
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. 2023. Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718.
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. 2023d, Side adapter network for open- vocabulary semantic segmentation.
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. 2022. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In Com- puter Eision-ECCV 2022: 17th European Confer- ence, Tel Aviv, Israel, October 23-27, 2022, Proceed- ings. Part XXIX, pages 736- -56,, Springer.
Yi Yang, Yueting Zhuang, and Yunhe Pan. 2021. Multi- ple knowledge representation for big data artificial in- telligence: framework, applications, and case studies. Frontiers of Information Technology Electronic Engineering, 22(12):1551-15


Reviews:
[{"review_id": "ckgkOvd8tf", "reviewer": "Reviewer_VMqU", "summary": "", "questions": "- This approach uses large models such as SAM, BLIP2 and CLIP. This might limit the applicability in the real-world use case. In this era where models pretrained on large-scale datasets are dominant, I am personally fine with that. But please mention your hardware configuration so that readers can correctly assess this method.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your precious questions. Here are our responses to your questions.\n****\n### The reviewer asked about hardware requirements and the inference latency.\n\n**Response:** Thank you for asking about computing resource requirements.\n\n**Hardware requirements**: Our method does not require high computing resources. Under the zero-shot setting, we do not finetune or train the models. All experiments were conducted on a single RTX 3090, which is applicable in real-world applications. The GPU memory consumption for the entire pipeline is about 22GB, including mask generation (SAM), captioner (BLIP2), and masked image-text matching (CLIP). \n\n**Inference latency:** We have tested the inference speed on a random selection of 500 images on a single RTX 3090. The CLIP-Only baseline method (mask generation + masked image-text matching)  obtains 1.88 seconds per image. The Global-Local method costs 2.01 seconds per image. Our method TAS (mask generation + captioner + masked image-text matching) achieves 3.63 seconds per image. \n\nGiven a fixed set of images to be referred to, TAS can be further accelerated by preprocessing image captions beforehand. By using the preprocessed captions, our TAS achieves 1.96 seconds per image and the GPU memory cost (mask generation + masked image-text matching) reduces to 11GB. Although introducing a captioning model inevitably extends inference time, it significantly improves the performance under the mIoU and oIoU metrics, e.g., +8.15 on oIoU and +7.05 on mIoU on the RefCOCO TestA set. The inference speed can be improved by using model quantization and other techniques. We will leave this as the future work of our work.\n\nWe will add a detailed inference speed comparison in the revision.\n\n***\nWe are grateful that the reviewer appreciates our work. Thank you for pointing out our typos. We will revise the writing carefully. We are happy to have more discussions about the paper if you have any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Performance is great! Nice work\n- Extensive ablation study and well-executed experiments", "reasons_to_reject": "- As this approach combines multiple models and requires (1) mining negative phrases (2) spatial rectifier, I wonder what is the inference cost compared to the baseline, as latency is also important.", "questions_to_authors": "- This approach uses large models such as SAM, BLIP2 and CLIP. This might limit the applicability in the real-world use case. In this era where models pretrained on large-scale datasets are dominant, I am personally fine with that. But please mention your hardware configuration so that readers can correctly assess this method.", "paper_topic_and_main_contributions": "This work leverage off-the-shelf SAM model and pretrained CLIP to handle zero-shot referring expression segmentation.\nSpecifically:\n- Clever combination of negative phrase together with S score (visual text matching) and P score (alignment quality from captioning model)\n- Strong zero-shot performance on widely-used benchmarks", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "Minor point but you should use \\citet instead of \\cite when the citation is used as a textual object, e.g. line 357"}, "extra_text": {}}, {"review_id": "exDGIPTFaN", "reviewer": "Reviewer_deYa", "summary": "", "questions": "Question A: While the performance gains on the Refcoco testB dataset are noteworthy, could you provide further results from other datasets to help validate the method's effectiveness and generalizability?\nQuestion B: In terms of transformer-based segmentation methods, particularly the recently popular Meta-SAM, could you outline the advantages of the proposed approach? It would be beneficial to highlight the unique strengths of your method.\nQuestion C: Visualization of instances where the method does not perform as expected might offer a valuable perspective on the current limitations. It could also help in identifying potential areas for improvement.\nQuestion D: Some parts of the paper could benefit from further refinement to enhance the readability and logical flow. For instance, it might be helpful to consolidate some sections in the methods part.\nI hope these queries will aid the authors in enhancing the strength of the paper. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your precious comments. We are very grateful that the reviewer appreciates our work. Our responses to the raised concerns are below.\n***\n### The reviewer is concerned about whether the comparison to other training methods (e.g., the weakly supervised training method TSEG) in the experiment is fair. \n\n**Response:** Thank you for the question. In the experiment section, we compared both weakly supervised methods (e.g., TSEG) and zero-shot methods (e.g., CLIP Surgey, Global-Local, CLIP-only) to demonstrate the effectiveness of our approach.\n\nCompared to weakly supervised training methods (e.g. TSEG), our method is more feasible and effective in real-world applications since it is training-free. Note that the settings of TSEG and our approach are different. TSEG is a weakly supervised method trained on the image-text pairs in the training set of RefCOCO, RefCOCO+, and RefCOCOg. It leverages the activation map toward expressions as the dense prediction results. **In contrast**, our approach follows the zero-shot setting, which does not require human-described referring expressions and is flexible with different off-the-shell models. Notably, our method outperforms TSEG even though no training data is leveraged. In addition, our method leverages SAM [1] for mask generation and SAM is trained on the SA1B dataset without accessing any COCO images. This indicates that TAS follows the zero-shot setting and it is capable of performing tasks without prior training or collecting specific examples. It is worth mentioning that the previous zero-shot method [2] also compared the weakly supervised method TSEG. We followed their setting and reported the result of TSEG as well. \n\nIn the revision, we will clearly specify the training setting of the methods (e.g., weakly-supervised or zero-shot).\n\n***\n### The reviewer asked about the performance gain in other datasets. \n**Response:** Thank you for the question. We have provided the ablation studies on the RefCOCO+ and RefCOCOg datasets. We adopted the VIT-B/32 CLIP backbone in the ablation studies.\n\nIn terms of the RefCOCO+ TestA set:\n|   $S_p$  |   $S_n$  |  Spatial |   oIoU   |   mIoU   |\n| :------: | :------: | :------: | :------: | :------: |\n|          |          |          |  33.74   |   43.73  |\n|  &check; |          |          |  36.72   |   47.25  |\n|  &check; |  &check; |          |  38.50   |   48.85  |\n|  &check; |  &check; |  &check; |  38.77   |   49.13  |\n\nIn terms of the RefCOCOg Umd validation set:\n|   $S_p$  |   $S_n$  |  Spatial |   oIoU   |   mIoU   |\n| :------: | :------: | :------: | :------: | :------: |\n|          |          |          |   30.66  |   42.66  |\n|  &check; |          |          |   32.41  |   44.50  |\n|  &check; |  &check; |          |   35.38  |   46.16  |\n|  &check; |  &check; |  &check; |   35.84  |   46.62  |\n\nThese results demonstrate the effectiveness of each proposed module. We hope that our response comprehensively addresses the concerns articulated by the reviewer. Thank you for pointing out the typos in our paper.\n***\n### References\n[1] Kirillov, Alexander, et al. \"Segment anything.\" arXiv preprint arXiv:2304.02643 (2023).\n\n[2] Yu, Seonghoon, Paul Hongsuck Seo, and Jeany Son. \"Zero-shot Referring Image Segmentation with Global-Local Context Features.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novelty: The paper proposes a new Text Augmented Spatial-aware (TAS) framework for zero-shot referring expression image segmentation. The framework consists of three main components: a mask proposal network, a text-augmented visual-text matching score, and a spatial rectifier. This approach introduces a novel way to leverage text information for image segmentation tasks.\n\n2. Performance Improvement: The paper demonstrates that the TAS framework outperforms previous methods in zero-shot referring expression segmentation. The proposed framework achieves higher accuracy and handles challenging scenarios such as occlusion and clustered objects effectively.\n\n3. Ablation Study: The paper includes an ablation study to analyze the effectiveness of each proposed module in the TAS framework. The results show that all modules contribute to performance improvement, and the spatial rectifier plays a vital role in handling orientation-related referring expressions.\n\n4. Experimental Evaluation: The paper conducts experiments and evaluation on multiple datasets, including RefCOCO, RefCOCO+, and RefCOCOg. The results demonstrate the superiority of the TAS framework over previous methods in terms of segmentation accuracy.\n\nOverall, the paper presents a framework for zero-shot referring expression image segmentation and provides thorough experimental evaluation and analysis. The results demonstrate the effectiveness of the proposed approach and its superiority over previous methods. Therefore, I recommend weakly accepting this paper.", "reasons_to_reject": "The proposed method is valuable and can effectively improve the performance of this task, but the authors use the Segmentation At Large Model (SAM) as priori knowledge model. Is this fair compared to other training methods (e.g., the weakly supervised training method TSEG) in the experiment?", "questions_to_authors": "Question A: While the performance gains on the Refcoco testB dataset are noteworthy, could you provide further results from other datasets to help validate the method's effectiveness and generalizability?\nQuestion B: In terms of transformer-based segmentation methods, particularly the recently popular Meta-SAM, could you outline the advantages of the proposed approach? It would be beneficial to highlight the unique strengths of your method.\nQuestion C: Visualization of instances where the method does not perform as expected might offer a valuable perspective on the current limitations. It could also help in identifying potential areas for improvement.\nQuestion D: Some parts of the paper could benefit from further refinement to enhance the readability and logical flow. For instance, it might be helpful to consolidate some sections in the methods part.\nI hope these queries will aid the authors in enhancing the strength of the paper. ", "paper_topic_and_main_contributions": "The paper discusses the task of zero-shot referring image segmentation, which aims to identify the instance mask that is most related to a referring expression without training on pixel-level annotations. The authors propose a Text Augmented Spatial-aware (TAS) framework that incorporates a mask proposal network, a text-augmented visual-text matching score, and a spatial rectifier for mask post-processing. The framework leverages a P-score and an N-score in addition to the visual-text matching score to improve alignment between referring sentences and local image regions. The proposed method outperforms state-of-the-art zero-shot referring image segmentation methods in experiments conducted on various datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 78: Rephrase \"its defects also obvious\" to \"its limitations are also apparent\" for better clarity.\nLine 61: Add \"the\" before \"early fusion framework\".\nLine 139: Change \"Endow the network\" to \"Endows the network\" to fix grammar error.\nLine 316: Simplify phrase \"obtained before\" to just \"obtained\"."}}, {"review_id": "AyVKYQhitD", "reviewer": "Reviewer_xQwB", "summary": "", "questions": "1. Given the significant differences in the RefCOCO, RefCOCO+, and RefCOCOg datasets, conducting ablation experiments solely on RefCOCO may not fully demonstrate the effectiveness of each module. It would be beneficial to show the performance on other datasets as presented in Table 4.\n\n2. The generated results from the Caption Model often differ greatly from the human-generated Reference. Moreover, images can be described in various ways, and the Negative Text Miner should handle this information correctly. Additionally, the reviewer raises concerns about how the Negative Text Miner deals with phrases in Reference and Caption that have different synonymous angles, such as \"a cute girl\" and \"young lady.\"", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful feedback. Here are our responses to your concerns.\n***\n### The reviewer mentioned that BLIP2 and GIT are very large models, which makes this method less practical. \n\n**Response:** Thank you for asking about computing resource requirements. It is indeed an important aspect of whether the method can be conducted in real-world scenarios.\n\n**For hardware requirements**, all experiments were conducted on a single RTX 3090. The entire pipeline (SAM+BLIP2+CLIP) consumes about 22GB GPU memory. It is feasible in real-world scenarios.\n\n**In terms of Inference Latency:**, we assessed inference speed using 500 random images on an RTX 3090. The CLIP-Only method (SAM+CLIP), results in 1.88 seconds per image. The Global-Local method (SAM+CLIP) costs 2.01 seconds per image. TAS (SAM+BLIP2+CLIP) achieves 3.63 seconds per image. \n\nTo save GPU memory and increase inference speed on a fixed dataset, we only run the captioning model once and store the generated captions for later use. Using the stored captions, the memory consumption reduces to 11GB. This is feasible even on an RTX 2080Ti. The inference time for a single image also decreases to 1.958 seconds.\n\nOverall, our method does not require high computing resources. Even though incorporating a caption model increases inference time, we obtain improvement in both oIoU and mIoU metrics, e.g., +8.15 oIoU, +7.05 mIoU on RefCOCO TestA set. Optimization methods like model quantization may further boost inference speed, we leave it as future work.\n***\n### The reviewer mentioned that the method presented in this article is quite similar to the Global-local approach.\n\n**Response:** Our method is different from previous methods. The main contribution of our work is designing a text-driven framework for better masked image-text matching. \n\nThe reviewer commented the $S_v$ was identical to previous works. Indeed, $S_v$ is widely used in previous works related to open vocabulary segmentation [1][2][3][4]. We clarified in our submission that this operation follows previous works (Line 269, Line 270).\n\n$S_p$ is different from the masked visual embedding introduced in the Global-Local paper. \nIn the Global-Local paper, the authors use the mask on the visual features to obtain global visual features. In contrast, we introduced a text-text matching score ($S_p$) based on the auxiliary description of each masked image for better region-text matching. This paradigm has not been studied in previous works related to zero-shot referring segmentation. Furthermore, the qualitative experiments in Figure 4 showed that masked visual embedding used in the Global-Local method favors larger masks, while our method is able to locate small objects.\n\n$S_n$ is novel in zero-shot referring segmentation. Motivated by repelling unrelated objects, we propose a negative text miner to extract unrelated expressions, enhancing the prediction by an elimination process. We first generate an overall caption for each input image. The overall caption summarizes all objects in the image. We then extract noun phrases from the caption using spacy and regard them as potential negative expressions. The similarity between masked images and negative expressions is termed as a negative score $S_n$. $S_n$ measures the probability of an object being unrelated to the referring expression. Results in  Table 4 demonstrate the effectiveness of $S_n$, e.g. +1.67 oIoU and +1.09 mIoU on RefCOCO.\n\nIn essence, our method is different from the Global-Local method in several aspects: \n1. **Flexibility with Model Architecture:** \nTAS is flexible with image-text matching models and captioning models. Table 6 and Table 7 in the paper show the versatility. In Table 6, using captions generated by the GIT-base model, TAS can still obtain +4.35 oIoU and +4.69 mIoU improvement on the RefCOCO+ TestA set. In Table 7, we substitude the image-text contrastive model CLIP by ALBEF and BLIP-2 and still observe a performance boost. Especially, we observed a surge of +17.37 oIoU and +23.4 mIoU on the ALBEF model.\n2.  **Integration of Text-Text Matching:**\nWe improved the typical masked image-text matching by incorporating descriptions of masked regions. We term the text-text matching score as $S_p$. Having $S_p$, we attain performance with respect to oIoU and mIoU. For instance, $S_p$ brings +2.43 oIoU and +2.93 mIoU in Table 4.  \n3. **Text-Centric Object Modeling:** \nTAS identifies unrelated image objects exclusively via negative text mining. The intuition is to filter distracting objects using negative expressions. The negative expressions are mined from the caption of the input images. We name the similarity between masked images and negative expressions $S_n$. As mentioned earlier, $S_n$ brings performance improvement as shown in Table 4.\n4. **TAS is spatial aware:**\nTAS is able to deal with spatial descriptions as we designed a spatial rectifier. The performance boost in Table 4 demonstrates the effectiveness.\n\n***\nOur responses to the questions from the reviewer are listed below:\n### Question 1. The reviewer asks about the ablation study on different datasets.\n**Response:** Thank you for the question. We have provided the ablation studies on the RefCOCO+ and RefCOCOg in the table below. Note that for the visual encoder, we adopt the VIT-B/32 CLIP backbone.\n\nIn terms of the RefCOCO+ TestA set:\n|   $S_p$  |   $S_n$  |  Spatial |   oIoU   |   mIoU   |\n| :------: | :------: | :------: | :------: | :------: |\n|          |          |          |  33.74   |   43.73  |\n|  &check; |          |          |  36.72   |   47.25  |\n|  &check; |  &check; |          |  38.50   |   48.85  |\n|  &check; |  &check; |  &check; |  38.77   |   49.13  |\n\nIn terms of the RefCOCOg Umd validation set:\n|   $S_p$  |   $S_n$  |  Spatial |   oIoU   |   mIoU   |\n| :------: | :------: | :------: | :------: | :------: |\n|          |          |          |   30.66  |   42.66  |\n|  &check; |          |          |   32.41  |   44.50  |\n|  &check; |  &check; |          |   35.38  |   46.16  |\n|  &check; |  &check; |  &check; |   35.84  |   46.62  |\n\nWe will add the ablation results in the final revision. Our results demonstrate the effectiveness of the proposed method. We observe the major improvement brought by $S_p$ and $S_n$, which validates our motivation of using text for better region-text alignment.\n\n### Question 2. The reviewer asked about how the Negative Text Miner deals with phrases in Reference and Caption that have different synonymous angles.\n**Response:** As mentioned in the $S_n$ section in the paper, we utilize Wordnet to judge whether the subject noun of the referring expression and the noun of each phrase extracted from the caption are synonyms. Specifically, we calculate the path similarity of the two synsets to determine whether to eliminate the synonyms. Empirically, we find the strict rules help TAS to identify distinct objects in these datasets. For instance, we believe \"young man\" and \"the boy\" are not synonyms. This involves the problem of semantic granularity, future work may investigate the influence of it. \n\n***\n### Further discussion on our motivation and idea\nWe are grateful the reviewer likes the task of zero-shot referring segmentation. Our intuition is to use text to enhance the region-text matching in a zero-shot manner. As the growth in works related to multi-modal areas, it is promising to directly integrate pretraining models[5][6] to solve different tasks. Considering the potential assistance between modalities[7][8], we investigate a text-driven manner for the zero-shot referring segmentation task without training or fine-tuning. Our method obtains state-of-the-art performance on the zero-shot referring expression segmentation task.\n\nIn summary, we hope our response addresses the concerns. These inquiries indeed mirror the challenges encountered during our research. We hope for a reconsideration of our submission. We believe the contribution our our paper matches the theme of the track. \n***\n### References\n[1]Xu, Mengde, et al. \"A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2]Subramanian, Sanjay, et al. \"Reclip: A strong zero-shot baseline for referring expression comprehension.\" arXiv preprint arXiv:2204.05991 (2022).\n\n[3]Jiang, Kenan, et al. \"Comclip: Training-free compositional image and text matching.\" arXiv preprint arXiv:2211.13854 (2022).\n\n[4]Liang, Feng, et al. \"Open-vocabulary semantic segmentation with mask-adapted clip.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[5]Zeng, Andy, et al. \"Socratic models: Composing zero-shot multimodal reasoning with language.\" arXiv preprint arXiv:2204.00598 (2022).\n\n[6]Shen, Yongliang, et al. \"Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.\" arXiv preprint arXiv:2303.17580 (2023).\n\n[7]Lin, Zhiqiu, et al. \"Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[8]Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PMLR, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In general, I really like this work because the task itself is valuable. Considering the lack of data for Referring Expression Segmentation, zero-shot methods have always been worth exploring.\n\nHowever, some major concerns remain.", "reasons_to_reject": "1. Both BLIP2 and GIT are very large models, which makes this method less practical. Especially for BLIP2, it is even challenging to directly run it on a V100 GPU. Therefore, if we use both SAM and BLIP2, only GPUs with capabilities like A100 or above might be able to handle this method. Additionally, the inference time will be very long, which further limits the real-world application of the method.\n\n2. The method presented in this article is quite similar to the Global-local approach. We can observe that the main difference between this article and Global-local lies in how they handle the local region. S^v is essentially identical to the original article, while S^p in the original article uses a mask on the visual encoding features, whereas this article uses a captioner. As for S^n, which is the main innovation in this article, I consider it not novel enough as similar methods are widely used in multimodal pre-training. Therefore, the contributions of this method seem somewhat incremental.\n\nI may raise the score if the above issues are reasonably addressed.", "questions_to_authors": "1. Given the significant differences in the RefCOCO, RefCOCO+, and RefCOCOg datasets, conducting ablation experiments solely on RefCOCO may not fully demonstrate the effectiveness of each module. It would be beneficial to show the performance on other datasets as presented in Table 4.\n\n2. The generated results from the Caption Model often differ greatly from the human-generated Reference. Moreover, images can be described in various ways, and the Negative Text Miner should handle this information correctly. Additionally, the reviewer raises concerns about how the Negative Text Miner deals with phrases in Reference and Caption that have different synonymous angles, such as \"a cute girl\" and \"young lady.\"", "paper_topic_and_main_contributions": "The author proposes a Referring Expression Segmentation method based on SAM+CLIP+BLIP2, using CLIP for matching and BLIP2 for filtering out irrelevant content in the image.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel zero-shot referring expression segmentation approach utilizing the SAM model, pretrained CLIP, and BLIP2. Pros include: (1) Strong performance on widely-used benchmarks, as highlighted by its zero-shot capabilities and innovative combination of visual-text matching and alignment quality scores. (2) Comprehensive ablation studies and well-conducted experiments that contribute to a deeper understanding of the method's efficacy. (3) A unique Text Augmented Spatial-aware (TAS) framework that offers new avenues for leveraging text information in image segmentation tasks. 
Cons comprise: (1) A sense of incrementalism, with some components of the proposed method bearing similarity to existing methods, thus questioning its true novelty. (2) Lack of clarity on the real-world applicability of the Negative Text Miner in handling diverse image descriptions and semantically varied phrases. After rebuttal and discussion, the AC feels that most concerns are addressed and this work is solid enough to appear in the conference but reviewers are generally less excited about it for the novelty concern.
Decision: Accept-Findings