Boosting Weakly Supervised Referring Image Segmentation via Progressive Comprehension
Authors: Zaiquan Yang
Yuhao LIU
Jiaying Lin
Gerhard Petrus Hancke
Rynson W. H. Lau
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-modal
segmentation
weakly-supervised
Primary area: machine_vision

Abstract: This paper explores the weakly-supervised referring image segmentation (WRIS) problem, and focuses on a challenging setup where target localization is learned directly from image-text pairs. 
We note that the input text description typically already contains detailed information on how to localize the target object, and we also observe that humans often follow a step-by-step comprehension process (\ie, progressively utilizing target-related attributes and relations as cues) to identify the target object. 
Hence, we propose a novel Progressive Comprehension Network (PCNet) to leverage target-related textual cues from the input description for progressively localizing the target object.
Specifically, we first use a Large Language Model (LLM) to decompose the input text description into short phrases. These short phrases are taken as target-related cues and fed into a Conditional Referring Module (CRM) in multiple stages, to allow updating the referring text embedding and enhance the response map for target localization in a multi-stage manner.
Based on the CRM, we then propose a Region-aware Shrinking (RaS) loss to constrain the visual localization to be conducted progressively in a coarse-to-fine manner across different stages.
Finally, we introduce an Instance-aware Disambiguation (IaD) loss to suppress instance localization ambiguity by differentiating overlapping response maps generated by different referring texts on the same image. 
Extensive experiments show that our method outperforms SOTA methods on three common benchmarks.

Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehension
Zaiquan Yang, Yuhao Liu', Jiaying Lin, Gerhard Hancke,, Rynson W.H. Lau
Department of Computer Science City University of Hong Kong
(zaiquyang2-c, yuhliug-c, mailiuuc,jiayinll (aapmnniinccm)mm cityciittccii.i .du. spp.aacke, Rynson Lau) ecityu. edu. hk
Abstract
This paper explores the weakly-supervised referring image segmentation (WRIS) problem, and focuses on challenging setup where target localization is learned directly from image-text pairs. We note that the input text description typically already contains detailed information on how to localize the target object, and we also observe that humans often follow step-by-step comprehension process (i.e., progressively utilizing target-related attributes and relations as cues) to identify the target object. Hence, we propose a novel Progressive Comprehension Network (PCNet) to leverage target-related textual cues from the input description for pro- gressively localizing the target object. Specifically, we first use Large Language Model (LLM) to decompose the input text description into short phrases. These short phrases are taken as target-related cues and fed into a Conditional Referring Module (CRM) in multiple stages, to allow updating the referring text embedding and enhance the response map for target localization in a multi-stage manner. Based the CRM, we then propose a Region- aware Shrinking (RaS) loss to constrain the visual localization to be conducted progressively in a coarse-to- fine manner across different stages. Finally, introduce an Instance-award Disambiguation (IaD) loss to suppress instance localization ambiguity by differentiating overlapping response maps generated by different referring texts the same image. Extensive experiments show that our method outperforms SOTA methods on three common benchmarks.
Introduction
Referring Image Segmentation (RIS) aims to segment target object an image via user-specified input text description. RIS has various applications, such as text-based image editing [17, 13, 2] and comannc interaction [62, 51]. Despite remarkable progress, most existing RIS works [7, 58, 27, 26, 21, 5] rely heavily on pixel-level ground-truth masks to learn visual-linguistic alignment. Recently, there has been surge interest developing weakly-supervised RIS (WRIS) methods via weak supervisions, egg., bounding-boxes [9], and text descriptions [54, 18, 30, 4], to alleviate burden of data annotations. In this work, we focus on obtaining supervision from text descriptions only.
The relatively weak constraint of utilizing text alone as supervision makes visual-linguistic alignment particularly challenging. There are some attempts [30, 18, 22, 46] to explore various alignment workflows. For example, TRIS [30] classifies referring texts that describe the target object as positive texts while other texts as negative ones, model a ext--to-mmage response map for locating potential target objects. SAG [18] introduces bottom-up and top-down attention framework to discover
Joint corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Qu "' player wearing blue and gray utiform cutches a bull"
Q1: player"
O Q2: "blue and gray uniform + O Q3: "catches ball"
(a) Image
(b) GT (Step-1)
(c) GT (Step-2)
(d) GT (Step-3)
(c) TRIS
(f) Ours (Step-1)
(g) Ours (Step-2)
(h) Ours (Slep-3)
Figure Given an image and language description as inputs (a), RIS aims to predict the target object (d). Unlike existing methods (e.g., TRIS [30] (e) a WRIS method) that directly utilize the complete language description for target localization, we observe that humans would naturally break down the sentence into several key cues (e.g., Q1 Q3) and progressively converge onto the target object (from (b) (d). This behavior inspires us to develop the Progressive Comprehension Network (PCNet), which merges text cues pertinent to the target object step-by-step (from (f) to (h)), significantly enhancing visual localization. @ denotes the text combination operation.
individual entities and then combine these entities as the target the referring expression. However, these methods encode the entire referring text single language embedding. They easily overlook some critical cues related to the target object in the text description, leading to localization ambiguity and even errors. For example, Fig. ((e), TRIS [30] erroneously activates all three players due to its use of cross-modality interactions between the image and the complete language embedding only.
We observe that humans typically localize target objects through a step-by-step comprehension process. Cognitive neuroscience studies [48, 41] also support this observation, indicating that humans tend to simplify a complex problem by breaking down into manageable sub-problems and reasoning them progressively. For example, in Fig. (b-d), human perception would first begin with "a player" and identify all three players (b). The focus is then refined by the additional detail "blue and gray uniform". ", which helps exclude the white player on the left (c). Finally, the action "catches ball" helps further exclude the person on the right, leaving the correct target person in the middle (d).
Inspired by the human comprehension process, propose in this paper novel Progressive Compre- hension Network (PCNet) for WRIS. We first employ a Large Language Model (LLM) [59] to dissect the input text description into multiple short phrases. These decomposed phrases considered as target-related cues and fed into a novel Conditional Referring Module (CRM), which helps update the global referring embedding and enhance target localization in multi-stage manner. We also propose a novel Region-aware Shrinking (RaS) loss to facilitate visual localization across different stages at the region level. ReS first separates the target-related response map (indicating the foreground region) from the target-irrelevant response map (indicating the background region), and then constrains the background response map progressively attenuate, thus enhancing the localization accuracy of the foreground region. Finally. we notice that salient objects in an image can sometimes trigger incorrect response map activation for text descriptions that aim for other target objects. Hence, introduce an Instance-aware Disambiguation (IaD) loss to reduce the overlapping of the response maps by rectifying the alignment score of different referring texts the same object.
In summary, our main contributions are as follows:
We propose the Progressive Comprehension Network (PCNet) for the WRIS task. Inspired by the human comprehension processes, this model achieves visual localization by progressively incorporating target-related textual cues for -ingul-in alignment.
Our method has three main technical novelties. First, we propose a Conditional Referring Module (CRM) to model the response maps through multiple stages for localization. Second, we propose a Region-aware Shrinking (RaS) loss to constrain the response maps across different stages for better cross-modal alignment. Third, to rectify overlapping localizations, we propose an Instance-aware Disambiguation (IaD) loss for different referring texts paired with the same image.

---

We conduct extensive experiments on three popular benchmarks, demonstrating that our method outperforms existing methods by a remarkable margin.
2 Related work
Referring Image Segmentation (RIS) aims to segment the target object from the input image according to the input natural language expression. Hu et al. [14] proposes the first CNN-based RIS method. There are many follow-up works. Early methods [60, 28, 38] focus object-level cross-modal alignment between the visual region and the corresponding referring expression. Later, many works explore the use of attention mechanisms [15, 7, 58, 19] or transformer architectures [58, 29] to model long-range dependencies, which can facilitate pixel-level cross-model alignment. For example, CMPC [15] employs a two-stage progressive comprehension model to first perceive all relevant instances through entity wording and then use relational wording to highlight the referent. In contrast, our approach leverages LLMs to decompose text descriptions into short phrases related to the target object, focusing on sentence-level (rather than word-level) comprehension, which aligns more closely with human cognition. Focusing on the visual grounding, DGA [56] also adopts multi-stage refinement. It aims to model visual reasoning on top of the relationships among the objects in the image. Differently, our work addresses the weakly RIS task and aims to alleviate the localization ambiguity by progressively integrating fine- fine-grained attribute cues.
-eakty-supervised RIS (WRIS) has recently begun to attract some attention, reduce the burden of data labeling especially on the segmentation field [25, 57, 63J. Feng et al. [9] proposes the first WRIS method, which uses bounding boxes for annotations. Several subsequent
can substantially
works [18, 22, 30] attempt to use weaker supervision signal, i.e., text descriptions. SAG [18] proposes first divide image features into individual entities via bottom-up attention and then employ top-down attention to learn relations for combining entities. Lee et al. [22] generate Grad-CAs for each word of the description and then consider the relations using an intra-chunk and inter-chunk consistency. Instead of merging individual responses, TRIS [30] directly learns the text-to-image response map by contrasting target-related positive and target-unrelated negative texts. Inspired by the generalization capabilities of segmentation foundation models [20, 8, 34J, PPT [6] enables effective integration with pre-trained language-image models [43,33] and SAM [20] by a lightweight point generator to identify the referent and context noise. Despite their success, these methods encode the full text as a single embedding for cross- mods-mod alignment, which overlooks target-related nuances in the textual descriptions. In contrast, method proposes to combine progressive text comprehension and object-centric visual localization to obtain better fine- mue-grained cross moosll aligmment.
Large I Language Models (LLMs) are revolutionizing various visual domains, benefited by their user- friendly interfaces and strong zero-shot prompting capabilities [3, 49, 47]. Building on this trend, recent works [42, 55, 53, 45, 64] explore the integration of LLMs into vision tasks (e.g., language- guided segmentation [55, 53], relation [23], and image classification [42]) through parameter-efficient fine-tuning or knowledge extraction. For example, LISA [55] and GSVA [53] utilize LLaVA [32], a large vision-language model (LVLM), as a feature encoder to extract visual-ling linguu-ic cross- modality features and introduce a small set of trainable parameters to prompt SAM [20] for reasoning segmentation. RECODE [23] and CuPL [42] leverage the knowledge in LLMs to generate informative descriptions prompts for different categories classification. Unlike these works, capitalize on the prompt capability of LLMs to help decompose a single referring description into multiple target object-related phrases, which are then used in our progressive comprehension process for RIS.
3 Our Method
In this work, observe that when identifying an object based on a description, humans tend to first pinpoint multiple relevant objects and then narrow their focus to the target through step-by-step reasoning [48, 41]. Inspired by this, we propose Progressive Comprehension Network (PCNet) for WRIS, which enhances cross-modalis modality alignment by progressively integrating target-related text cues at multiple stages. Fig. 2 shows the overall framework of our PCNet.
Given an image and a referring expression T as input, we first feed T into a Large Language Model (LLM) to break it down into K short phrases Teub (lo, 41, ..., 4x-1). referred to as target-related text cues. We then feed image and referring expression T and the set of short phrases
Toub into image encoder and text encoder to obtain visual feature Vo E R R WXC. and language
3

---

IaD
Propesal Generator
Ro
Rue
B
Ra-li
Innge-
T
R
Ch
player wouring blue gray uniform
Image Encoder
Response Map Generation
Response Map Generation
Respomic Map Generation
D
In context Instruction
LLM
Va-1
Referring Modulaatio
Referring Modudation
Modialation
playeer wears blunigny anform catcher ball
Text Encoder
Teui
On
Qest
Clamilication
Heginn-anare Shrinking
Ieft
Instane--naa Ulcamtiguaaiion
CIM
Conditonal Reteriing Madal:
+
Conconmaaion
Figure 2: The pipeline of PCNet. Given a pair of image-text as input, PCNet enhances the visual- linguistic alignment by progressively comprehending the target-related textual nuances in the text description. starts with using L.IM to decompose the input description into several target-related short phrases target-related textual cues. The proposed Conditional Referring Module (CRM) then processes these cues to update the linguistic embeddings across multiple stages. Two novel loss functions, Region-aware Shrinking (RaS) and Instance-aware Disambiguation (IaD), are also proposed to supervise the progressive comprehension process.
feature Qo RIxC. and Quub {qo- q1 ... 9K-1), with qu E RRIC. where H H1/s s and W = Wi/s. C and Cj denote the numbers channels of visual and text features. s is the ratio of down-sampling. We then projector layers transform the visual feature Vo and textual features Qo and @rah to unified dimension C, ie., Vo E ER H XC Qo and q are in RIxC
We design PCNet with multiple consecutive Conditional Referring Modules (CRMs) progressively locate the target object across N stages' Specifically, at stage n, n-th CRM updates the referring embedding Qn into Qa+1 conditioned the short phrase qn from the proposed Referring Modulation block. Both Qu+1 and visual embedding V,, are fed into Response Map Generation to generate the text-to-image response map R, and updated visual embedding V n++ Finally, the response map Rv generated by the n-th CRM is used as the final localization result. To optimize the resulting response map for accurate visual localization, to obtain localization mitttcheennmmteheed matched mask proposals. We also propose Region aware Shrinking (RaS) loss constrain the visual localization in coarse-to-fine manner, and Instance-aware Disambiguation (IaD) loss to suppress instance localization ambiguity.
employ the pre- preitrained proposal generator
In the following subsections, we first discuss how we decompose the input referring expression into target-related cues in Sec. 3.1. We then introduce the CRM in Sec. 3.2. Finally, we present our Region-aware Shrinking loss in Sec. 3.3, and Instance-aware Disambiguation loss in Sec. 3.4.
3.1 Generation of Target-related Cues
Existing works typically encode the entire input referring text description, and easily overlook some critical cues (e.g., attributes and relations) in the description (particularly for a long/complex description), leading to target localization problems. To address this problem, we propose dividing the input description into short phrases to process it individually. To do this, we leverage the strong in-context capability of the L.M [1] to decompose the text description. We design a prompt, with four parts, to instruct the LLM to do this: (1) general instruction Pc: (2) output constraints Pc; (3) co-tontext task examples g and (4) input question PQ. P& describes the overall instruction, egg. "decomposing the referring text into target object-related short phrases"" Pe elaborates the output setting, e.g., sentence length each short phrase. P F, we specifically curate several in-context pairs guidance for the L.M to generate analogous outputs. Finally, PQ encapsulates the input text description and the instruction words for the L.M execute the operation. The process of generating
Note that all counts

---

target-related cues is formulated as:
Toub {fo. f1, ..,, tk-1) LLM (Pc. Pc, Po)
(I)
where K represents the total number of phrases, which varies depending on the input description. Typically, longer descriptions more likely yield more phrases. To maintain consistency dataset, we standardize to five phrases (i.e., K = 5). If fewer than five phrases are produced, we simply duplicate by LLM are related to the target object and align closely with our objective.
training
of the short phrases obtain five short phrases. In this way, phrases generated
3.2 Conditional Referring Module (CRM)
Given the decomposed phrases (i.e., target-related cues), we propose CRM to enhance the dis- criminative ability on the target object region conditioned on these phrases, thereby improving localization accuracy. As shown in Fig. the CRM operates across N consecutive stages. At each stage, it first utilizes a different target-related cue to modulate the global referring embedding via referring modulation block and then produces the image-to-texx response map through response map generation block.
Referring Modulation Block. Considering the situation at stage n,, first concatenate one target- related cue On and the L negative text cues obtained from other images 2, to form q/, [ We then fuse the visual features V, with q,, through a vision-to-text cross-attention, to obtain as:
vision-attended cue features qn E R(I-1)xC
A SoftMax (a', & (V.W))T/VV)) ( qn = MLP @ (V. + 9n
where A.-. E B((++))xX> denotes the vision-to-text inter-modality attention weight. W and W are learnable projection layers. @ denotes matrix multiplication. Using the vision-attended cue features qn. we then enrich the global textual features Qn into cue-enhanced textual features Qn+1 E _IxC through another text-to-text cross-altention, as:
A SoftMax ((Q, W @ (3) (T Qa+1 MLP(A @ (q, W3)) Qn.
(3)
where Ar E RIx(++I) represents the text-to-text intra-mode moddlity attention weight. /.º and W are learnable projection layers. In this way, we can enhance the attention of Q, on the target object by conditioning its own target-related cue features and the global visual features.
Response Map Generation. To compute the response map, we first update the visual features V, to V, by integrating them with the updated referring text embedding Q++1 using text-to-visual cross-attens atts-attention, thereby reducing the cross moddaltyy discrepancy. Note that V,, is then used in the next stage (i.e., Vn41 V,). The response map R., E T H at the n-th stage is computed as:
R₂ = Norm Norm(Rel.u(V, s QT+1)))
where Norm normalizes the output in the range of [0,1]. To achieve global visual- linguliin alignment, we adopt classification loss Lass in [30] to optimize the generation of the response map at each stage. formulates the target localization problem as a classification process to differentiate between positive and negative text expressions. While the referring text expressions for an image are used as positive expressions, the ones from other images can be used as negative for this image. More explanations given in appendix.
3.3 Region-aware Shrinking (RaS) Loss
Despite modulating the referring attention with the target-related cues stage-by-stage image-text classification often activates irrelevant background objects due to its reliance on global and coarse response map constraints. Ideally, as the number target-related cues used increases across each stage, the response map should become more compact and accurate. However, directly constraining the latter stage to have a more compact spatial activation than the former stage can lead to a trivial
Refer to the Appendix for more details.
5

---

solution (i.e., without target activation). To address this problem, we propose a novel region aware shrinking (RaS) loss, which segments the response map into foreground (target) and background (non-target) regions. Through contrastive enhancement between these regions, our method gradually reduces the background interference while refining the foreground activation in the response map.
Specifically, at stage n, we first employ pretrained proposal generator to obtain a set of mask proposals, .M- {mi, m2, .... mp), where each proposal mp is in R#* and P is the total number of segment proposals. We then compute a alignment score between the response map R, and each proposal my in M as:
(5) where @ denotes the hadamard product. The proposal with the highest score (denoted as my) is then treated the target foreground region, while the combination of other proposals (denoted as my)
Sn (sn.1 Sn.2. 1hr, Pn with Sn.p = max (R,, ① mp):
is regarded as non-target background regions. With the separated regions, we define a localization ambiguity Sumb which measures the uncertainty of the target object localization the current stage n, as:
Sumb 1 - my IO,, (b))).))..
(6)
where Sumbb is in the range of [0, 1J. and IoU denotes the intersection over union. When the localization result (i.e. the response map) matches the only target object proposal instance exactly, ambiguity is O. Conversely, if it matches the more background proposals, ambiguity approaches 1.
Assuming that each target the image corresponds to an instance, by integrating more cues, the model will produce a more compact response map and gradually reduce the ambiguity. Consequently, based on the visual localization results from two consecutive stages, we can formulate the region-aware shrinking objective for a total of N stages as:
N 2
CRaS N
max (0, (Sar1 - -S (amb))
==0
By introducing region-wise ambiguity. Lass can direct non-target regions to converge towards attenuation while maintaining and improving the quality of the response map in the target region. This enables the efficient integration of target-related textual cues for progressively finer cross-modal alignment. Additionally, the mask proposals can also provide a shape prior to the target region, which helps to further enhance the accuracy of the target object localization.
3.4 Instance-aware Disambiguation (IaD) Loss
Although the RaS loss can help improve the localization accuracy by reducing region-wise ambiguity within one single response map, it takes less consideration of the relation between different instance- wise response maps. Particularly, different objects of an image, there are usually some overlaps among the corresponding response maps. For example, in Fig. 2, the player in the middle is simultaneously activated by two referring expressions (i.e., the response maps R., and Re. have overlapping activated regions), resulting in inaccurate localization. To address this problem, we propose an Instance-aware Disambiguation (IaD) loss to help enforce that different regions of the response maps within a stage are activated if the referring descriptions of an image refer to different objects.
note that, given different referring deseriptions that refer to
Specifically, given a pair of image I, and input text description T,, we first sample extra Na text descriptions, Ta = {t1. tz. ..., ). where the referred target object of each text description ta is in the image I, but is different from the target object referred to by Ta. We then obtain the image-to-ter response maps R., and R. (R1, R2, NN,, } for T. and Ta through Eq. (4). Here, we omit the stage index n for clarity. Then, based on the Eq. (5), we obtain the alignment scores S, and (Sa) 1= the corresponding proposal (specified by the index) and the current text. To disambiguate overlapping activated regions, we constrain that the maximum index of the alignment score hetween S, and each of SI must be different from each other (i.e., different texts must activate different objects). Here, we follow [50] to compute the index vector, y E 12 x1 as:
Na for Tr and Ta. In S, the larger the value, the higher the alignment between
y one hotehho(eee whw-toottrreeee hot(argmax(()) S -sg(S),
(8)

---

Table 1: Quantitative comparison using mloU and PointM metrics. "(U)" and "(G)" indicate the UMD and Google partitions. "Segmentor" denotes utilizing the pre-trained segmentation models (SAM [20] by default) for segmentation mask generation. denotes that the method is fully-supervised. "_" means unavailable values. Oracle represents the evaluation of the best proposal mask based ground-truth. Best and second-best performances marked in bold and underlined.
RefCOCO
RefCOCO+
RefCOCOg
Metric
Method
Rackhone
Segmentor
Val
TestA
TestB
Val
TestA
TestB
Val (G)
Val (U)
Test (U)
GiT [54]
GrT
x
25.0
26.3
24.4
25.9
26.0
26.1
30,0
30,9
31.0
CLIP-ES [25]
ViT-Base
x
41.3
50.6
30.3
46.6
56.2
33.2
46.2
45.8
WWbL [46]
VGG16
x
31.3
31.2
30.8
34.5
33.3
36.1
29.3
32.1
31.4
PointM
SAG [18]
ViT-Base
x
56.2
63.3
51.0
45.5
52.4
36.5
37.3
TRIS [30]
ResNet-50
x 51.9
60.8
43.0
40.8
40.9
41.1
52.5
51.9
53.3
PCNetr
ResNet-50
x 59.6
66.6
48.2
54.7
65.0
44.1
57.9
$7.0
57.2
PCNets
ResNet-50
x
60.0
69.3
52.5
58.7
65.5
45.3
58.6
57.9
57.4
LAVT'
Swin-Base
N/A
72.7
75.8
68.7
65.8
70.9
59.2
63.6
63.3
63.6
GroupViT [54]
GroupViT x
18.0
18.1
19.3
18.1
17.6
19.5
19.9
19.8
20.1
CLIP-ES [25]
ViT-Base
x
13.8
15.2
12.9
14.6
16.0
13.5
14.2
13.9
14.1
TSRG [15]
ViT-Small
x
25.4
22.0
22.1
WWbL
VGG16
x
18.3
17.4
19.9
19.9
18.7
21.6
21.8
21.8
21.8
SAG [18]
ViT-Base
x 33.4
33.5
33.7
28.4
28.6
28.0
28.8
mloUt
TRIS [30]
ResNet-50
x 25.1
26.5
23.8
22.3
21.6
22.9
26.9
26.6
27.3
PCNet,
ResNet-50
30,9
35.2
28.9
31.9
26.5
29.8
29.7
30.2
PCNets
ResNet-50
x
31.3
36.8
26.4
29.2
32.1
26.8
30.7
30.0
30.6
CLIP [43]
ResNet-50
36.0
37.9
30.6
39.2
42.7
31.6
37.5
37.4
37.8
SAG [18]
ViT-Base
44.6
50.1
38.4
35.5
41.1
27.6
23.0
TRIS [30]
ResNet-50
41.1
48.1
31.9
31.6
31.9
30.6
38.4
39.0
39.9
PPT [6]
ViT-Base
46.8
45.3
46.3
45.3
45.8
44.8
43.0
PCNets
ResNet-50
52.2
58.4
47.9
56.5
36.2
47.3
46.8
46.9
Oracle
ResNet-50
72.7
75.3
67.7
73.1
75.5
68.2
69.0
68.3
68.4
where sg(+) represents the stop gradient operation. Finally, we denote the index vectors for S, and { Na d=1 as ya and d=1" [ and we formulate the IaD loss as:
1 Na
CIOD
1laa yall?).
By enforeing the constraint each stage, the response maps activated by different referring descrip- tions in image for different instances are separated, and the comprehension of the discriminative cues is further enhanced.
Experiments
4.1 Settings
Dataset. We have conducted experiments on three standard benchmarks: RefCOCO [61], RefCOCO+ [61], and RefCOCOg [39]. They are constructed based on MSCOCO [24]. Specially, the referring expressions in RefCOCO and RefCOCO+ focus tively, and they are characterized by succinct descriptions, averaging 3.5 words in length. RefCOCOg contains much longer sentences (average length of 8.4 words), making it more challenging than others. RefCOCOg includes two partitions: UMD [40] and Google [39].
object positions and appearances, respec-
Implementation Details. We train our framwork for 15 epochs with batch size of 36 on an RTX4090 GPU. The total loss for training is Listal = Lera + Laas LTaD- By default, we set number of stages N to 3, and the number of the additional text descriptions sampled for each image N₂ I. Without loss of generality, we use FreeSOLO [52] and SAM [20] as the proposal generators to obtain two versions: PCNets and PCNetp. Refer to Sec. A for more implementation details.

---

"sus squatting head pour "Iuse Referring Expression;; Jensoon the batker and suuire"
Referring Expressions:
julry blakk
"a uniig
(n)
(b)
Referring Expressions:
"asull stadlagg
"i shawp eaiing gros"
Referring Expressions:
vananing
(d)
Figure VVisual results of our PCNet. The green markers denote the peaks of the response maps.
Evaluation Metrics. We argue that the key to WRIS is target localization, and the performance evaluation should not rely primarily on pixel-wise metrics. With the accurate localization points, pixel-Ievel masks can be readily obtained by prompting the pre-trained segmentors (e.g.. SAM [20]). Thus, following [18, 22, 30J, we adopt localization-bet -aased metric (i.e., PointM), and pixel-wise metrics (i.e., mean and overall intersection-over-unio (mloU and oloU) for evaluation. PointM [30] is used to evaluate the localization accuracy, which computes the ratio of activation peaks in the mask region.
4.2 Comparison with State-of-the-Art Methods
Quantitative comparison. Tab. 1 compares our method with various SOTA methods. Specifically, we first compare target localization accuracy using the PointM metric. We evaluate two model variants: P and PCNets, which use different segmentors (e.g.. FreeSOLO [52] and SAM [20]) to extract mask proposals for RaS and IaD losses. Even when using FreeSOLO as the proposal generator, our model still significantly outperforms all compared methods. For example, on the most challenging dataset, RefCOCOg, with more complex object relationships and longer texts, pCNet achieves performance improvements of 55.2% and 10.3% on the Val (G) set compared to SAG and TRIS'. PCNets further boosts the performance if we replace FreeSOLO with the stronger SAM.
In addition, we verify the accuracy of the response map through pixel-wise mIoU metric. Results shown in the middle part of Tab. Our PCNet still achieves superior performances on all benchmarks, against all compared methods. Particularly, PCNet, and PCNets outperform TRIS by an improvement of 10.8% and 14. 1% mloU, respectively, on the RefCOCOg Val (G) set. In the bottom part of Tab. 1, we compare the accuracy of the extracted mask proposals generated using the target localization point (i.e., the peak point of the response map) to prompt SAM. We can see that our PCNet significantly outperforms other WRIS methods. We can also see that higher PointM values correlate with higher mloU accuracy values of the corresponding mask proposals for different methods. We further tested the mask accuracy using the Ground-Trutt localization point (i.e. the last row), and find that its performance even surpasses the fully aully-supervised method, LAVT [58]. All these results highlight the critical importance of target localization (i.e., peak point) for the WRIS task.
In Fig. we show some visual results of our method across different scenes by using the target localization point (i.e., the green marker) to prompt SAM to generate the target mask. Our method effectively localizes the target instance among other instances within the image, even in complex scenarios with region occlusion (a), multiple instances (b), similar appearance (c), and dim light (d).
4.3 Ablation Study
We conduct ablation experiments on the RefCOCOg dataset and report the results on the Val (G) set from both PCNets and PCNet in Tab. 2, and from PCNet in Tab. 3, Tab. 4, and Tab. 5.
Component Analysis. In Tab. 2, first construct single-stage baseline (Ist row) to optimize desual-linguistee alignment by removing all proposed components and using only the global image- text classification loss Lcie. We then introduce the proposed conditional referring module (CRM) to the baseline to allow for multi-stage progressive comprehension (2nd row). To validate the efficacy of the region-aware shrinking loss (RaS) and instance-aware disambiguation loss (IaD), we introduce them separately (3rd and 4th rows). Finally, we combine all proposed components (5th row).
For a fair comparison, remove its 2nd stage as it is used for enhancing pixel-wise mask accuracy.
8

---

Table 2: Component ablations on RefCOCOg Val (G) set.
h
PointM PCNets PCNete
mloU
oloU
Cas CRM Lus Ciu
PCNets PCNet
PCNels PCNet
51.7
25.3
25.1
53.3
26.8
26.7
57.7
56.4
29.8
28.5
29.6
28.5
55.3
54.3
28.3
27.7
28.2
27.8
/
58.6
57.9
30.7
29.8
30.6
30.1
Table 3: Ablation of the num- ber of iterative stages N.
Table 4: Ablation of different modulation strategies CRM.
Table 5: Ablation of the num- bers descriptions Na IaD.
N
mloU
oloU
PointM
Method
mloU
oloU
PointM
Na
mloU
oloU
PointM
1
27.4
27.3
55.3
ADD
28.5
28.4
56.3
28.5
28.5
56.4
2
29.3
29.4
57.3
29.3
29.1
57.1
29.8
30.1
57.9
29.8
30.1
57.9
VTA+ADD
29.2
29.1
57.2
29.8
29.7
57.8
29.5
29.8
56.7
VTA+TTA
29.8
30.1
57.9
29.7
29.6
57.7
The results demonstrate that even using only LeIa, progressively introducing target-related cues through CRM can still significantly enhance target object localization. In particular, PCNets achieves improvements of 3.1 % on PointM and 5.9% on mloU; @ by using LRaS to constrain response maps, making them increasingly compact and complete during the progressive comprehension process. the accuracy of target localization is dramatically enhanced, resulting in an improvement of 11.6% on PointM. although LID can facilitate the separation of overlapping response maps between different instances within the same image and improve the discriminative ability of our model on the target object, the lack of constraints between consecutive stages results in smaller performance improvement than LRas: and ? all components are essential for our final PCNet, and combining them achieves the best performance. two examples. We can see that each component can bring obvious localization improvement.
Fig. 4, we also provide the visual results of the ablation study on
Number of Iterative Stages. In Tab. 3, analyze the effect of the number of iterative stages N. When N = 1, we can only apply Las and LLDD, but not Lass, resulting in inferior results. Increasing N from to 2 significantly improves the performance due to the progressive introduction of target- related cues. However, the improvement from N = 2 to N = 3 is less pronounced than from N = 1 to 2, and the performance stabilizes N 3. N 4, the performance slightly declines. This is because when the effective short-phrases decomposed by L.MM are fewer than the number of stages, we need to repeat text phrases later stages, which may affect the loss optimization.
Modulation Strategy. In Tab. 4, we ablate different variants of CRM: directly adding target cue features qu and global referring features Q,, (denoted as ADD): @ fusing qa and Q,. using only text-to-text cross-attention (denoted as TTA); 2 first employing a vision-to-text cross-attention fuse visual features V, and qи obtain vision-atee atteeddd features qn. and then adding them to Qn (denoted VTA+ADD). The results demonstrate that ADD is the least efficient method. TTA
Q: "a light brown color sweet vada with dark brown one next to it"
Q: "a catcher rushing to make a play on the hall"
Image
(b) Less
(c)+ CRM
(d) Caus
(e) Cran
GT
Figure 4: Visualization of the ablation study to show the efficacy of each proposed component.

---

Q "3 tookly bears sitting on bod"
(a) Image
(b) GT
(c) Ours
Figure 5: A failure case of our PCNet. As our model design assumes that there is only one object referred to by the language expression, it usually returns only one object.
outperforms ADD but is less effective than VTA+ADD, verifying the importance of the vision context. Finally, our CRM combines VTA and TTA and achieves the best results.
Number of Referring Texts. In Tab. 5, analyze the effect of Na used in LLAD- The results show that N₂ = 1 is enough, and the performance deteriorates as N increases. This is because an image typically has 2-3 text descriptions, which means Na should be 1-2. As Na increases, repeated sampling becomes more frequent, affecting model training and thus leading to poorer results.
Conclusion
In this paper, have proposed a novel Progressive Comprehension Network (PCNet) perfom progressive visnal-linguistic alignment for the weakly referring image segmentation (WRIS) task. PCNet first leverages LLM decompose the input referring description into several target-related phrases, which are then used by the proposed Conditional Referring Module (CRM) to update the referring text embedding stage-by-ster we proposed two loss functions, region-avee aware shrinking loss and instance- disambiguation loss, facilitate comprehension of the target-related cues progressively. We have also conducted extensive experiments on three RIS benchmarks. Results show that the proposed PCNet achieves superior visual localization performances and outperforms existing SOTA WRIS methods by large margins.
thus enhancing target localization. In addition.
Our method does have limitations, For example, shown in Fig. 5, when the text description refers to multiple objects, our method fails to return all referring regions. This is because our model design always assumes that there is only one object referred by the language expression. In the future, we plan to incorporate more fine-gra grained vision priors [35, 44] and open- open-world referring descriptions (e.g., camouflaged [12], semi- transparanp [31], shadow [36] and etc.) into the model design to enable more generalized solution.
References
[1] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, Aleman, F.L., Almeida, D., Al- tenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv:2303.08774 (2023)
[2] Avrahami, O., Lischinski, D., Fried, O.: Blended diffusion for text-driven editing of natural images. In: CVPR. pp. 18208-18218 (2022)
[3] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al.: Language models few-shot learners. pp. 1877-1901 (2020)
[4] Chen, D., Wu, Z., Liu, F., Yang, Z., Huang, Y., Bao, Y., Zhou, E.: Prototypical contrastive language image pretraining. arXiv preprint arXiv:2206. 10996 (2022)
[5] Chng, Y.X., Zheng, H., Han, Y., Qiu, x., Huang, G.: Mask grounding for referring image segmentation. arXiv:2312.12198 (2023)
[6] Dai, Q. Yang, S.: Curriculum point prompting for suaely-s referring image seg- mentation. In: Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13711-13722 (2024)
10

---

[7] Ding, H., Liu, C., Wang, S., Jiang, X.; Vision-language transformer and query generation for referring segmentation. In: ICCV. pp. 16321-16330 (2021)
[8] Du, Y. Bai, F. Huang, T., Zhao, B.: Segvol: Universal and interactive volumetric medical image segmentation. arXiv preprint urXiv:2311.13355 (2023)
[9] Feng, G., Zhang, L.. Hu, Z., Lu, H.: Learning from box annotations for referring image segmentation. IEEE TNNLS (2022)
[10] He, C., Li, K. Zhang, Y, Xu, G., Tang, L.,, Zhang, Y., Guo, Z., Li, X.: Weakly-supervised concealed object segmentation with sam-based pseudo labeling and multi-scale feature grouping. NeurlPS 36 (2024)
[11] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770-778 (2016)
[12] He, R., Dong. Q., Lin, J., Lau, R.W.: Weakly-superviser camouflaged object detection with scribble annotations. In: AAAI (2023)
[13] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-or, D.: Prompt-to-prompt
image editing with cross-attention control. In: ICLR (2022)
[14] Hu, R., Rohrbach, M., Darrell, T.: Segmentation from natural language expressions. In: ECCV. pp. 108-124 (2016)
[15] Huang, S., Hui, T., Liu, S., Li, G., Wei, Y., Han, J., Liu, L.,, Li, B.: Referring image segmentation via cross-modal progressive comprehension. In: CVPR. pp. 10488-10497 (2020)
[16] Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.I,, Bressand, F. Lengyel, G., Lample, G., Saulnier, L., et al.: Mistral 7b. arXiv preprint arXiv:2310.06825 (2023)
[17] Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., Irani, M.: Imagic: Text-based real image editing with diffusion models. In: CVPR. pp. 6007-6017 (2023)
[18] Kim, D., Kim, N., Lan, C., Kwak, S.: Shatter and gather: Learning referring image segmentation with text supervision. In: ICCV. pp. 15547-15557 (2023)
[19] Kim, N., Kim, D., Lan, C., Zeng, W., Kwak, S.: Restr: Convolution-fie referring image segmentation using transformers. In: CVPR. pp. 18145-18154 (2022)
[20] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: ICCV. pp. 4015-4026 (2023)
[21] Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., Jia, Lisa: Reasoning segmentation via large language model. arXiv:2308.0068 (2023)
[22] Lee, J., Lee, S., Nam, J., Yu, S., Do, J., Taghavi, T. Weakly supervised referring image segmentation with intra-chunk and inter-chunk consistency. In: ICCV. pp. 31870-21881 (2023)
[23] Li. L., Xiao, J., Chen, G., Shao, J., Zhuang, Y., Chen, L.: Zero-shot visual relation detection via composite visual cues from large language models. In: NeurIPS (2024)
[24] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: ECCV, pp. 740-755 (2014)
[25] Lin, Y., Chen, M., Wang, W., Wu, B., Li, K., Lin, B., Liu, H., He, X.: Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation. In: CVPR. pp. 15305-15314 (2023)
[26] Liu, C., Ding, H., Jiang, X.: Gres: Generalized referring expression segmentation. In: CVPR. pp. 23592-2360) 23601 (2023)
[27] Liu, C., Ding, H., Zhang, Y., Jiang, X.: Multi-modal mutual attention and iterative interaction for referring image segmentation. IEEE TIP (2023)

---

[28] Liu, D., Zhang, H.. Wu, F., Zha, Z.J.: Learning to assemble neural module tree networks for visual grounding. In: ICCV. pp. 4673-4682 (2019)
[29] Liu, F., Kong. Y., Zhang. L., Feng. G., Yin, B.: Local-glone global coordination with transformers for referring image segmentation. Neurocomputing (2023)
[30] Liu, F., Liu, Y., Kong, Y., Xu, K., Zhang, L.. Yin. B., Hancke, G., Lau, R.: Referring image segmentation using text supervision. In: ICCV. pp. 22124-22134 (2023)
[31] Liu, F., Liu, Y., Lin, J., Xu, K., Lau, R.W.; Multi-view dynamic reflection prior for video glass surface detection. In: AAAI (2024)
[32] Liu, H., Li, C., WU, Q., Lee, YJ.:: Visual instruction tuning (2024)
[33] Liu, Y., Wang, X., Zhu, M., Cao, Y., Huang, T., Shen, C.: Masked channel modeling for
bootstrapping visual pre-training. International Journal of Computer Vision pp. 1-21 (2024)
[34] Liu, Y.. Zhu, M., Li, H., Chen, H., Wang, X., Shen, C.: Matcher: Segment anything with one shot using all-purpose feature matching. arXiv preprint arXiv:2305.13311 (2023)
[35] Liu, Y., Ke, Z., Liu, F., Zhao, N., Lau, R.W.: Diff-plugin: Revitalizing details for diffusion-based low-level tasks. In: CVPR (2024)
[36] Liu, Y., Ke, Z., Xu, K., Liu, F., Wang, Z.,, Lau, R.W.: Recasting regional lighting for shadow removal. In: AAAI (2024)
[37] Loshchilov, I,, Hutter, F: Decoupled weight decay regularization. In: ICLR (2018)
[38] Luo, G., Zhou, Y., Sun, X., Cao, L.,, Wu, C., Deng, C., Ji, R.: Multi-task collaborative network for joint referring expression comprehension and segmentation. In: CVPR. pp. 10034-10043 (2020)
[39] Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A.L., Murphy, K.: Generation and comprehension of unambiguous object descriptions. In: CVPR. pp. 11-20 (2016)
[40] Nagaraja, V.K., Morariu, V.I., Davis, L.S.: Modeling context between objects for referring expression understanding. In: ECCV. pp. 792-807 (2016)
[41] Plass, J.L.,, Moreno, R., Brünken, R.: Cognitive load theory. Cambridge University Press (2010)
[42] Pratt, S., Covert, I., Liu, R., Farhadi, A.: What does a platypus look like? generating customized prompts for zero-shot image classification. In: ICCV. pp. 15691-15701 (2023)
[43] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748-8766 8763 (2021)
[44] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022)
[45] Salewski, L., Alaniz, S., Rio-Torto, I., Schulz, E., Akata, Z.: In-context impersonation reveals large language models' strengths and biases. In: NeurlPS (2024)
[46] Shaharabany, T., Tewel, Y., Wolf, L.: What is where by looking: phrase-grounding without text inputs. pp. 28222-28237 (2022)
open-world
[47] Shao, H., Qian, S., Xiao, H., Song, G., Zong. Z., Wang, L., Liu, Y., Li, H.: Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models. arXiv preprint arXiv:2403. 16999 (2024)
[48] Simon, H.A., Newell, A.: Human problem solving: The state of the theory in 1970. American Psychologist (1971)
[49] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al.: Llama: Open and efficient foundation language models. 13Xiv:2302.13971 (2023)
12

---

[50] Van Den Oord, A., Vinyals, O., et al.: Neural discrete representation learning (2017)
[51] Wang, X., Huang, Q. Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.F. Wang, W.Y., Zhang, L.: Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In: CVPR. pp. 6629-6638 (2019)
[52] Wang, X., Yu, Z.,, De Mello, S., Kautz, J., Anandkumar, A., Shen, C., Alvarez, J.M.: Freesolo: Learning to segment objects without annotations. In: CVPR. pp. 14176-14186 (2022)
[53] Xia, Z., Han, D., Han, Y., Pan, X., Song, S., Huang, G.: Gsva: Generalized segmentation via multimodal large language models. urXiv:2312.10111 (2023)
[54] Xu, J. De Mello, S., Liu, S., Byeon, W., Breuel, T., Kautz, J., Wang, X.: Groupvit: Semantic segmentation emerges from text supervision. In: CVPR. pp. 18134-18144 (2022)
[55] Yang, S., Qu, T., Lai, X., Tian, Z., Peng, B., Liu, S., Jia, J.: An improved baseline for reasoning segmentation with large language model. arXiv:2312.1722 17240 (2023)
[56] Yang. S., Li, G., Yu, Y.: Dynamic graph attention for referring expression comprehension. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 4644 4653 (2019)
[57] Yang, Z., Ke, Z., Hancke, G., Lau, R.: Cross-domain semantic decoupling for weakly-supervised semantic segmentation (2023)
[58] Yang. Z., Wang. Tang, Y., Chen, K. Zhao, H., Torr, P.H.: Lavt: Language-aware vision
transformer for referring image segmentation. In: CVPR. pp. 18155-1816 (2022)
[59] Ye. Chen, X., Xu, N., Zu, C., Shao, Z., Liu, S., Cui, Y., Zhou, Z., Gong, C., Shen, Y., et al.: A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv:2303. 10422 (2023)
[60] Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., Berg, T.L.: Matmet: Modular attention
network for referring expression comprehension. In: CVPR. pp. 1307-1315 (2018)
[61] Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: ECCV. pp. 69-85 (2016)
[62] Zhu, F., Zhu, Y., Chang, X.. Liang, X.: lasiuaelanguage navigation with self-supervised auxiliary reasoning tasks. In: CVPR. pp. 10012-10022 (2020)
[63] Zhu, L., Zhou, J., Liu, Y., Hao, X., Liu, W., Wang, X.: Weaksam: Segment anything meets
weakly-supervised instance-level recognition. arXiv preprint arXiv:2402.14812 (2024)
[64] Zong, Z., Ma, B., Shen, D., Song. G., Shao, H., Jiang, D., Li, H., Liu, Y.: Mova: Adapting mixture vision experts to multimodal context. arXiv preprint arXiv:2404. 13046 (2024)
13

---

A More Implementation Details
A.1 Generation of Target-related Cues
To obtain multiple target-related cues, we leverage the strong in-context capability of the Large Language Model (LLM) [16] to decompose the input referring expression and ob- tain the target-related textual cues. The Fig. 6 presents the LLM prompting details.
The prompt includes four parts:
(I) general instruction Pc. (2)
player wearing blue and uniform catches hall
Referring text describes one objectt scene with attributes information, hope you cun enumerate the state attribute of the abject. output sentence include and the first should given the referred object. For example, given the referiig text: woman white shirt looking down laptop computer. should output: I. woman. 2. woman in white woman looking
output constraints Pc, (3) in-
context task examples P Eg and (4) imput question Pq. In part Pc. we define a overall instruc- tion for our task (i.e, decompos- ing the referring text) Then in part Pc, we elaborate some de- tails about the output (e.g, the sentence length for each cue de- scription). part P E. we curate several in-context learning examples guidance for the LLM to generating analogous output. Con- sidering that the input referring expressions contain various sentence structures, in part Pg the more examples given, the more reliable the output will be. The part Pq instructs the LLM to output the results given the input referring expression.
LLM
Fext clnes:
He wears blueignay uniform He cutches ball
player
Then given the text D Give your output:
computer.
Figure 6: Flow of LLM-based referring text decomposition.
(b)
(c)
Q: person wet suit stands in water
Q: mag half full of beer sitting on the left of
animal sticking face through
la person].
fence
[The person in wet suit]
A: [a mugl
A: [an animal]
person water]
[The mag half bocr]
[an animal sticking ils face through the fence]
[Tee mag the left of the table]
woman in black shit with her hands
red shiet with
small napkin cupeake
under her chin
hand by her face A: [a woman]
(a small sopica] [The sapkin with cupcake printed it] small sapkin]
woman]
[The weman in a blsck shirt]
shirt]
hands under clinn]
[The woman with her hand by her face]
Q: front left borse.
couch left.
Q: dog at top of picture.
horse]
[a couch]
A: (a dog]
[The bose in frout left]
[TTe couch in left)
dog top picture]
Figure 7: LLM generated examples. We show the LLM generated examples for long language expressions in (a)-()) and the ones for short language expressions in (g)-(i). "Q" denotes the input language expression and the "A" denotes the output relaet textual cues of LLM.
14

---

In Fig. 7, we give some examples of decomposing the referring expressions. The "Q" denotes the imput referring content and the "A" denotes the answers of LLM. In most cases, we can obtain reliable target-related textual cues that do not contradict the original text input. Besides, we also notice that there are some cases in which the text not sufficiently decomposed (e.g, the example (c)) or the LLM outputs redundant results (e-g, the example (f)), which hinders the model to benefit from progressive comprehension to some extent.
A.2 Text-to Image Classification Loss
Our work consists of multiple stages and utilizes Leie in TRIS [30] at each stage independently for response maps optimization. Here, we omit the index of stage n for clarity. Leie formulates the target localization problem as a classification process to differentiate between positive and negative text expressions. The key idea of Lels loss function is contrast image-text pairs such that correlated image-text pairs have high similarity scores and uncorrelated image-text pairs have low similarity scores. While the referring text expressions for an image are used as positive expressions, the referring text expressions from other images can be used as negative expressions for this image. Thus, given batch (i.e., B) image samples, each sample is mutually associated with one positive reference text (i.e..a text describing specific object in the current image) and mutually exclusive with L negative reference texts (texts that are not related to the target object the image). Note that the number of batches is equal to the sum of the positive samples and the negative samples (i.e., B = + L).
Speficially, in each training batch, B image-text pairs (I, T are sampled. Through the language and vision encoders, B xW xc Then, we obtain the response maps R E B by applying cosine similarity calculation and normalization operation. After the pooling operation as done in TRIS, we further obtain the alignment score matrix y E R BB According to the Leis, for in image in the batch, there a prediction score yli. :]. where y(i, i] predicted by the corresponding text deserves a higher value (i.e, the positive one) and the others deserve lower values (L negative ones). Then classification loss for the 4h image from the batch can be formulated as cross-entropy loss:
can get referring embeddings Q BBx and image embeddings V
Letes
Σ j=1
11= log (1 (1+e-xta))
)
(1 11= log
1 (1te)tta))
e e-Hlll
B
and the classification loss for the batch can be formulated as:
1 1
Lesa
Cass,
B
The i denotes the index for the visual image and the j denotes the index for the referring text.
A.3 Referring Modulation Block
In Sec. 3.2, we have mentioned that the conditional referring module (CRM) utilizes the decomposed textual cues to progressively modulate the referring embedding via a modu- lation block across N consecutive stages, and then produces the image-to-text response map by computing the patch-based similarity between visual and language embeddings. Specifically, the modulation block is im- plemented by a vision-to-text and a text- to-text cross-attention mechanism in cas- cade for facilitating the interaction between cross-modal features. In Fig. 8, we give an overview of the block design. For the block each stage, We concatenate one target- related cue and the L. negative text cues obtained from other images as the condi- tional text cues and then obtain the vision- attended cue features by the vision-to-text attention. Then by learning the interaction between referring embedding and different textual cue embeddings, the block is expected enhance the integration of discriminative cues.
Qu+1
V.
A
q
A
Linear layer
Figure 8: Illustration of referring modulation block.
15

---

A.4 Training and Inference
We implement our framework on PyTorch and train it for 15 epochs with batch size 36 (i.e..
L + 1) on a RTX4090 GPU with 24GB of memory. The imput images are resized to 320 x 320. We use ResNet-50 [11] as our backbone of image encoder, and utilize the pre-trained CLIP [43] model to initialize the image and text encoders. The down-sampling ratio of visual feature = 32, the channels of vision feature C. = 2048, text features C 1024, and the unified hidden dimension C = 1024. The network is optimized using the AdamW W optimizer [37] with weight decay of le-2 and an initial learning rate of 56-5 with polynomial learning rate decay. For the L.M,, we utilize the open-source powerful language model Mistral 7B [16] for referring text decomposition. For the proposal generator, we set the number of extracted proposals P = 40 for each image.
B More Quantitative Studies
B.1
Comparisons with other SOTA methods
Table 6:
Different criterions for alignment
Table 7: Different criterions for alignment score measurement in LIAD-
score measurement in Lass-
Alignment Score
mloU
PointM
oloU
Alignment Score
mloU
PointM
oloU
Max
29.8
57.9
30.1
Max
29.8
57.9
30.1
Avg
29.1
56.4
29.2
Avg
28.8
54.9
29.0
In Tab, 6 and Tab. 7, we conduct the abla- tion studies about the measurement crite- rion of alignment score. The results demon- strate that the maximum value of the re- sponse map in each proposal better repre- sents the alignment level of region-wise cross-modal alignment than the average value. To validate the effectiveness of the modeling the progressive comprehension, we also quanti- tatively compare the outputs of our method at different stages in Tab. 8. The results show that the localization results gradually improve with more discriminative cues integration, especially in the early stages.
Table 8: Comparison between different stages.
Stage Num.
Stage 0
Stage
Stage
mloU
28.6
29.4
29.8
PointM
56.7
57.6
57.9
B.2 More Ablation Studies
Comparison between IaD loss and others. In our IaD loss LISD. we adopt a hard as- signment for deriving the loss function as GiT [54]. The motivation is that we aims to get the pseudo mask prediction by the accurate peak value point (i.e., the hard assign- ment results) instead of relying on whole score distribution S(.) (e.g. Sa, Sa in Sec. 3.4). Thus utilizing the hard assignment to de- rive the IaD loss well matches our purpose, which helps rectify the ambiguous localiza- tion results. If we use the soft assignment (e.g., measuring KL divergence between S. and Sz). though the equivalent may be simpler, it not only does not match our pur- pose but also introduces more tricky com- ponents for optimization (c.g., extra distribution regularization is required). In order to verify the argument, we conduct comparison RefCOCOg(G) val dataset as Tab. 9. The LIaD even causes slight decline, while the proposed loss LRL brings clear improvement on localization accuracy.
Table 9: Comparison between IaD loss and KL loss.
Lata
Cx
Lia
PointM
mloU
$1.7
25.3
1
51.2
24.8
53.1
26.6
Comparison between IaD loss and calibration loss in TRIS. There are essential differences between them. First, the calibration loss in TRIS [30] is used to suppress noisy background activation and thus help to re-calibrate the target response map. In contrast, in our method, we observe that, multiple referring texts corresponding to different instances one image may locate the same instances (or say overlapping), due to the lack of instance-level supervision in WRIS.
16

---

Table 10: Comparison between IaD loss and calibration
As for the implementation. the calibra- tion loss adopts the global CLIP score of image-text to implement simple con- trastive learning for revising the response map. Differently, we simultaneously in- fer the response maps of different referring texts from the same image, and obtain the instance-level localization results by choos-
loss Leal
Lets
LIaD
Lea1
PointM
mloU
50.3
24.6
51.4
26.4
54.7
26.3
ing the mask proposal with max alignment score. To further verify the superiority of our loss, conduct an ablation on the RefCOCO(val) dataset. We use the TRIS without calibration loss as the baseline and then separately introduced these two loss functions for comparison. Both ablations demonstrate that the laD loss not only refines the response map (mloU) but also significantly improves the localization accuracy (PointM).
C More Visualization of Localization Results
Progressive Comprehension for Localization. In Fig. 9, we also give visualization of each stage's response map for qualitative analysis. The results show that our proposed CRM module effectively integrate the target-related textual cues. For example, in the first row, the method produces ambiguous localization result the first stage. After taking the cue "with gold necklace" " into consideration, the attention is transferred to the target object at the second stage. Finally, after considering all the cues, the method produces less ambiguous and more accurate localization results.
Q: woman with gold necklace sitting behind little birthday girl
Q1: woman Q: base ball player kneeling down with his co-player
Q2: She is with...
Q3: She behind
Q1: a base ball player Q2: He kneels down... Q: a young man with shirt that has giant musical note on it
Q3: NULJ
Q1: young man Q: boy in black t-shirt and jeans bending by keeping his hands on knees
Q2: He with shirt
Q3: The shirt has
Q1: a boy
Q2: a boy in black...
Q3: The boy is bending
Figure 9: Visualization of progressive localization. With the integration of discriminative cues, the identification of target instance gradually improves.
17

---

Qualitative Comparison with Other method. More qualitative comparisons of our method with other methods are shown the Fig. 10. For the example shown in the fifth row, the query is "a man in arm striped sweater". The TRIS [30] mistakenly locates the left man as the target regions. In contrast, our PCNet optimizes the response map generation process by continuously modulating the referring embedding query conditioned on the target-related cues instead of static referring embedding. As result, our method can obtain more accurate localization result.
D Proposal Generator
In this work, adopt the representative pre-trained segmentors: FreeSOLO [52] and SAM [20] for proposal generation. Specifically, the FreeSOLO [52] is a fully unsupervised learning method that learns class-agnostic instance segmentation without any annotations. The SAM's training utilizes densely labeled data, but it does not include semantic supervision. This supervision does contradict weakly supervised RIS setting. More importantly, offers promising solution as an image segmentation foundation model and can be used for refining the coarse localization results from
methods into precise segmentation masks as done the recent works [10, 63]. For the usage of SAM, We adopt the ViT-H backbone, the hyperparameter predicted iou threshold and stability score threshold are set to 0.7, and points per side is set 8.
In Fig. 11 and Fig. 12, we give the generated mask proposals examples by FreeSOLO [52] and SAM [20], respectively. We notice that there often overlaps among the generated proposals. Thus, we refine the generated proposals by filtering out candidate proposals with small area (the threshold set 1000) and then selecting the ones with smaller intersection over union (the threshold is as 0.8). Considering that the number of proposals generated by the segmentor may be different for maintain consistency by selecting the top 40 proposals with the largest area (P = 40). If fewer than 40, we simply complete it with an all-zero mask.
different image inputs, in implementation,
E Broader Impacts
While we do not foresee our method causing any direct negative societal impact, may potentially be leveraged by malicious parties to create applications that could misuse the segmentation capabilities for unethical illegal purposes. We urge the readers to limit the usage of this work legal use cases.
18

---

Q: "the guy wearing white sitting on the couch watching his friends play video games"
Q: "a catcher rushing to make play on the ball"
Q: "a boy in black 1 shirt and jeans bending by keeping his hands on knees"
Q: "a smiling man with a small infant wearing a beige - shirt and gray jeans" T
Q: "a man in arm striped sweater"
Q: "a postcard with picture of face of cute girl"
Q: "dark haired woman wearing a blue jacket next to teddy bear"
(a) Image
(b) TRIS
(c) SAG
(d) PCNet
(e) GT
Figure 10: More visual comparison between our method with TRIS and SAG for WRIS.
19

---

COCO 1000_1 train2014 0000 1000122800
COCO train2014 00001000011
GOOD Sainznn 1000 WOTADAAI
Figure 11:
Free [52] generated mask proposals examples.
20

---

COCO train2014 000000233661
COCO train2014 000000110015
COCO train2014 000000157191
O0
Figure 12: SAM [20] generated mask proposals examples.
21

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In Sec. 1, we make the claims to reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Sec. 5, we discuss the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide implementation details in Sec. and the Appendix for reproducibil-
ity.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [No]
Justification:
We will consider releasing the data and code once the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nip) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtppcccpppccceeccccccccccccccat / ssssssolicyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide details about the experimental setting (e.g., hyperparameters) in Sec. 4 and the Appendix for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We do not report error bars due to limited computing resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information on the computer resources the implementation details of Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / /wwuurruusp...e cc cctppblic//eeeeccccccccccceceeee wttellii///iiiieeeeeee cethcchhueiieeeeees
Answer:
[Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses societal impacts of the work in the Appendix See, E.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly cites the existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "UlKfEpst8m", "reviewer": "Reviewer_oLx4", "summary": "This paper proposes a Progressive Comprehension Network for weakly-supervised referring image segmentation, which mimics the human process of progressive understanding by breaking down sentences into segments and gradually narrowing down the target range. The main contributions include: A multi-stage Conditional Referring Module to progressively comprehend text cues. A Region-aware Shrinking Loss to constrain the target region to gradually shrink. An Instance-aware Disambiguation Loss to eliminate overlap between different instances.", "questions": "1. Why is the range of Equation 6 set to [0,1]? Can the combination of other proposals (denoted as 𝑚_𝑏 ) be simply approximated by the complement of the foreground, since the proposals segmented by SAM nearly cover the entire image? In this case, if IoU(𝑅_𝑛,𝑚_𝑓) < IoU(𝑅_𝑛,𝑚_𝑏), wouldn't Equation 6 be greater than 1?\n2. Does 𝑚_𝑓 change at different stages in Equation 7, for instance, when different proposals are selected at different stages? What issues could this cause? In Equation 7, are 𝑛 and 𝑛+1 reversed? According to Section 3.3, the ambiguity score should be lower at later stages.\n3. Section 3.1 mentions that there are five phrases for each sentence, and Section 3.2 states that each stage uses one text cue, so there should be five stages. Why do the implementation details mention only three stages?\n4. In Section 3.4, how do you sample extra 𝑁_𝑑 texts? Are they randomly sampled from multiple text descriptions corresponding to the same image? How do you ensure that the sampled text descriptions correspond to different target referents? Because in refcoco series dataset, one referent may have multiple text expressions.\n5. In the bottom part of Tab.1, are the numbers all obtained by using a single peak point as the prompt for SAM? Did you try using the response map as the prompt for SAM?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Presentation: The majority of the paper is well-written and easy to follow, providing clear explanations of the key concepts.\n2. Motivation: The motivation is reasonable and convincing. By mimicking the human process of understanding concepts from coarse to fine, the method progressively refines the final mask. Directly requiring the mask to continuously shrink could lead to trivial solutions, and two losses are introduced to further support this pipeline. The proposed method aligns well with the motivation.\n3. Experiments: Comprehensive ablation experiments demonstrate the effectiveness of the proposed modules", "weaknesses": "1. The concept of mimicking the human process of understanding concepts from coarse to fine-grained, along with multi-stage refinement for different parts of sentences, has been explored in prior REC research [1]. This approach involves parsing sentences into multiple parts and conducting multi-stage refinement. Applying REC's ideas [1] to WRIS can also be considered a contribution, but it should be thoroughly discussed in comparison to [1], highlighting any differences, and cited appropriately.\n2. There is a lack of discussion and citation of recent WRIS work, such as [2], in related work and the main text. It is necessary to discuss and cite the latest WRIS work in both the related work section and the main table.\n\n3. Section 3.4 is not entirely understandable. I suggest the authors re-describe this section to make it clearer and easier to understand.\n  \n4. The top of Fig.2 is not clear enough, especially the subscripts a and d, I suggest the author re-design this part.\n\n\n [1] Yang S, Li G, Yu Y. Dynamic graph attention for referring expression comprehension[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 4644-4653.\n\n [2] Dai Q, Yang S. Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13711-13722.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincerely thanks for useful comments. To the weaknesses and questions, our response is as follows:\n\n**[W1]**: \n\nThank you for your advice. Although both DGA [A] and our method adopt multi-stage refinement, there are significant differences: \n- The motivation is different. DGA focuses on the fully-supervised visual grounding task and aims to model visual reasoning on top of the relationships among the objects in the image. In contrast, our work addresses the weakly RIS task and aims to alleviate the localization ambiguity by progressively integrating fine-grained attribute cues. \n- The implementation is different. In the absence of mask, we build the relation between the response map of different stages using the proposed  RaS loss.\n- We will discuss and include their differences in the revision.\n\n**[W2]**:\n\nThank you for your advice. PPT [B], appearing on Arxiv near the NeurIPS deadline, is considered contemporaneous work. It introduces a parameter-efficient point prompting framework. The Table below presents a comparison on the val sets using **mIoU**. While our method utilizes down-sampled SAM mask proposals, we also tested with full-resolution masks for fairness, aligning with PPT. In both scenarios, our method demonstrates superior target localization compared to PPT. We will discuss  and include their differences  in the revision.\n\n| Methods                      | val (RefCOCO) | val (RefCOCO+) | val (RefCOCOg_Google) |\n| -|:-:|:-:|:-:|\n| PPT                          | 46.7          | 45.3           | 42.9                  |\n| Ours (down-sampled SAM mask) | **48.6**      | 43.3           | **45.4**              |\n| Ours (raw SAM mask)          | **52.2**      | **47.9**       | **48.3**              |\n\n**[W3 & W4]**:\n- Explanation about `Sec. 3.4`\n  - **Motivation**. Considering that each image generally contains multiple instances, each paired with its reference description, the weak discriminative capability of the model in WRIS often leads to multiple texts for different instances in a single image activating the same region (i.e, response map overlap in Line-209). Thus, we propose the **IaD** loss in `Sec. 3.4`  to encourage referring texts for different instances (in the same image) to activate different regions. \n  - **Implementation**. Since this loss function is applied individually to each image sample, for clarity, we omitted the batch concept and consider only one sample, and omitted the subscript `a` used in the Line 215-Line 227. Specifically, given one image-text pair $\\\\{\\mathbf{I}, \\mathbf{T}\\\\}$, we sample $N\\_d$ (By default,  $N\\_d=1$)  texts for current image sample, which means that we have two referring texts (i.e., $\\mathbf{T}$ and $\\mathbf{T}\\_{d}$ )  for the image $\\mathbf{I}$. Then, we can obtain response maps $\\mathbf{R}$ and $\\mathbf{R}\\_{d}$ according to the description in `Sec 3.2`, and get the alignment scores $S$ and $S\\_{d}$ according to `Eq. (5)`. Afterwards, we get the instance indexes $\\texttt{argmax}(S)$ and $\\texttt{argmax}(S\\_{d})$ as the proposal predictions by them.  Considering $\\texttt{argmax}(\\cdot)$ is a non-differentiable operation during gradient backward, we adopt a differentiable implementation by `Eq. (8)`. Finally, we can impose MSE loss on the indexes to encourage the two texts (in the same image) to activate different regions by `Eq (9)`.\n- We will revise the figure to make the subscripts clear in the revision.\n\n**[Q1]**:\n\n- Thanks for pointing out this issue. This is a typo, and the range shold be [0,2], as in some cases, it may occur that $\\text{IoU}(𝑅_𝑛,𝑚_𝑓) < \\text{IoU}(𝑅_𝑛,𝑚_𝑏)$.\n\n- It is less feasible. The shrinking loss constraint the response map toward the compact orientation while maintaining located in the foreground area.  $𝑚_𝑏$ include less foreground part and more background than the $m_f$. It would lead to more background activation when choosing the $𝑚_𝑏$ as the complement of the foreground.\n\n**[Q2]**:\n\nIt is possible to have different instances for different stages. According to our statistical result, this issue occurs in a small number of cases (less than 10%) and has minimal impact on loss optimization. We will include this analysis  and revise the Eq. (7) in the revision. \n\n**[Q3]**:\n\nLLM outputs for referring texts often vary in attribute cue count (2-5). To enable parallel training, we standardize them to 5 cues via repetition padding (`Sec. 3.1`). The Tab. 3's ablation study on stage numbers shows optimal performance with 3 stages, thus our framework's implementation utilizes three stages.\n\n**[Q4]**:\n\nIn addition to the standard training batch ($B$ image-text pairs), we sample $N_d$ extra texts ($N_d=1$) per image. Leveraging the **refer_id** and **image_id** annotations in refcoco datasets, which link texts to unique instances, we randomly sample texts referring to different instances from the original.\n\n**[Q5]**:\n\nAll results utilize a single peak point as the SAM prompt. As suggested, we compared different prompts on the validation set (see table below). Using the response map as the prompt proved sensitive to thresholds, impacting noise ratios and yielding inferior performance compared to the single peak point prompt.\n\n| Methods                      | val (RefCOCO) | val (RefCOCO+) | val (RefCOCOg_Google) |\n| -| :-:|:-:|:-:|\n| single peak + SAM            | **52.2**      | **47.9**       | **48.3**              |\n| pseudo mask (thre=0.8) + SAM | 37.7          | 33.4           | 38.4                  |\n| pseudo mask (thre=0.5) + SAM | 41.1          | 35.6           | 39.3                  |\n| pseudo mask (thre=0.2) + SAM | 39.4          | 35.3           | 35.6                  |\n\n**References**\n\n[A] Dynamic graph attention for referring expression comprehension--ICCV 2019\n\n[B] Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation--CVPR. 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wdbmfnPZVk", "reviewer": "Reviewer_kTSb", "summary": "Inspired by human's step-by-step cognitive process for localizing a target object in an image, the paper proposes Progressive Comprehension Network (PCNet) for the task of weakly-supervised referring image segmentation (WRIS) where text is the only supervision signal. PCNet first decomposes a long, complex text description into multiple target-related cues using a large language model (LLM), and the specialized designed module, Conditional Referring Module (CRM), progressively refines the text-to-image response map using the generated text cues in multiple stages. Region-aware Shrinking (RaS) is introduced for constraining the latter stage to have a more compact, target-related response map, and Instance-aware Disambiguation (IaD) loss is proposed for differentiating between the target object and the other objects in the input image. The experiment results show the superiority of the proposed method.", "questions": "Please rebut the above weaknesses. I also have minor questions:\n- In Figure 2, the arrow comes from Q_{n+1} to Cls. How is Q_{n+t} used for computing the classification loss?\n- How did the authors sample L negative textual cues? one cue from each of the examples in the same mini-batch?\n- I am curious about the rationale behind Equations (2) and (3). Why did the authors use residual connection only after the mlp layer, not also after the self-attention layer like in Transformer?\n- Does the order in which text cues are used affect the performance of the model? E.g., is there a performance difference between feeding into the CRM module in the order of q_0, q_1, q_2 and in the order of q_0, q_2, q_1?", "limitations": "The paper addressed the limitations of the work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The paper provides extensive experiment results proving the validity of the proposed method, PCNet, and also ablation studies showing the contributions of main components of the model.\n- Figures help readers understand the proposed method.", "weaknesses": "- The proposed method uses the contrastive loss of TRIS, but not generates independent response maps from a positive cue and L negative cues. In Equation (3) and (4), it integrates all the text cue features and generates a single response map R_n that contains information from both positive and negative cues. This does not make sense. How did the authors differentiate between positive and negative cues when computing the classification loss L_Cls? (This is why I rated \"poor\" for presentation).\n- IaD, the loss to differentiate between the target object and other objects in the same image is similar to the calibration loss used in TRIS; the idea is the same while the calibration loss uses CLIP image-text similarity for differentiation. The authors should have discussed about this loss in the paper, and also done an ablation study comparing the two losses.\n- The PCNet uses the mask proposals generated by a pre-defined segmentation mask generator such as FreeSOLO and SAM. We cannot ignore the possibility of knowledge being distilled from the mask generator into PCNet, especially through RaS and IaD losses, main contributions of the paper. More specifically, we cannot be sure that the performance gains from using RaS and IaD do not come from the mask generator's knowledge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincerely thanks for your useful comments. To the weaknesses and questions, our response is as follows:\n\n**[W1]**:\n\nThank you for you careful comment. We sincerely apologize for the less detailed presentation regarding the  loss  $\\mathcal{L}_{\\texttt{cls}}$ in **TRIS** [25] due to space restrictions. We provided a detailed  explanation of the idea and implementation of this loss in the **general response**.  We will include this loss explanation in the revision.\n\n**[W2]**:\n\nThe idea of two loss functions is different and there are  two essential differences between them:\n\n- **Motivation**. The calibration loss in TRIS is used to suppress noisy background activations and thus help to re-calibrate the target response map.  In contrast, in our method, we observe that, multiple referring texts corresponding to different instances in one image may locate the same (or we say overlapping) instances (described in `lines 210-212`), due to the lack of instance-level supervision in WRIS. \n\n- **Implementation**. The calibration loss adopts the global CLIP score of image-text to implement a simple contrastive learning for revising the response map. Differently, we simultaneously infer the response maps of different referring texts from the same image, and obtain the **instance-level** localization results by choosing the  mask proposal with max alignment score. `IaD` can achieves better localization accuracy with instance-level differentiation.\n\n\n- To further verify the superiority of our loss, we conduct **two-groups ablations** on the **RefCOCO (val)** dataset. The first ablation used `PCNet` without `IaD` loss as the baseline, and  the second used the TRIS without calibration loss. We then separately introduced these two loss functions  for comparison. Both ablations demonstrate  that the `IaD` loss not only refines the response map (mIoU metric) but also significantly improves the localization accuray (PointM metric). \n\n  - baseline  (CLS+CRM+RaS)\n\n    | Baseline | Calibration loss | IaD loss | PointM   | mIoU  |\n    | - | :-:|:-:|:-:|:-:|\n    | &#10003; |  |  | 56.1 | 28.7   |\n    | &#10003; | &#10003;  |  | 56.9     | 29.9   |\n    | &#10003; |  | &#10003; | 60.0 | 31.3 |\n\n  - baseline (CLS)\n\n    | Baseline | Calibration loss | IaD loss | PointM | mIoU |\n    | - | :-:|:-:|:-:|:-:|\n    | &#10003; |   |  | 50.3   | 24.6 |\n    | &#10003; | &#10003;  |  | 51.4 | 26.4 |\n    | &#10003; |  | &#10003; | 54.7 | 26.3 |\n\n**[W3]**:\n\nWe would address your concerns from the following four aspects:\n\n- Mask generator does not contain any semantic information. Though it can be used as post-processing method to improve the completeness of the mask, here we aims to improve our localization accuracy by introducing suitable constraint based on the mask proposals.\n\n- Our core idea is to improve the understanding of textual cue information by progressive process. The ablation experiments (Table 2) demonstrate that even without **RaS** and **IaD** loss, our approach improves localization to some extent. However, relying only on global classification loss is suboptimal due to its lack of modeling at the instance level.  Therefore, we introduce **RaS**, which leverages mask proposals to model multi-stage refinement and enhance instance-level cross-modal alignment.\n\n- In lines `200-204`, we point out that a complete mask proposal can improve the mask quality to some extent, but it is not the core of our `RaS` loss.  In the following, we conducted a quantitative analysis on **RefCOCO** val set. \n\n  - The first line denotes the results of baseline (include classification loss and CRM module). In the second line, we utilize the mask proposal generated by SAM (refer to `line-189`) as pseudo mask to calculate  the `IoU` loss between it and the response map. The third line denotes the results of our `RaS` loss. \n  - While the guidance of the mask proposal can improve the accuracy of the response map to some extent, there is a significant gap with the `RaS` loss on the localization performance. Besides, as shown in Table 1, even with the less accurate mask proposal by FreeSOLO, our methods still achieve superior localization performance. These results show that the performance improvement of our method is not exactly due to the introduction of mask generator.\n\n  | Methods    | PointM   | mIoU   |\n  | - | :-:|:-:|\n  | Baseline (CLs +CRM)  | 53.5   | 26.4   |\n  | Baseline  + IoU_loss | 54.5   | 27.9   |\n  | Baseline  + RaS      | 56.1 | 28.7 |\n\n- In addition, the `IaD` loss only utilizes the mask proposal as a localization result to derive the loss formula, there is no exploitation of mask knowledge.\n\n**[Q1]:**\n\nI guess the question concerns how $Q_{n+1}$, not $Q_{n+t}$ (absent in Fig. 2), is used for calculating the Classification loss. As explained in the **general response**, $Q_{n+1}$ in stage $n$ contains one positive and $L$ negative referring embeddings per image. Given this known text-image correspondence, computing the cross-entropy loss is straightforward.\n\n**[Q2]:**\n\nLeveraging the **refer_id** and **image_id** annotations in datasets, which link texts to unique instances, we randomly sample texts referring to different instances from the original. Simultaneously the cues for each text in the batch is sampled (one cue is used for referring embedding modulation at each stage). \n\n**[Q3]**: \n\nAbout the residual design, we draw inspiration from the previous works (like TRIS[25] and DenseCLIP[A]). The design can integrate useful cross-modal information while preserving the original modal information from degradation.\n\n**[Q4]**:  \n\nThe order of the text inputs would not distinctly affect the performance of our method. In the table below, we conducted a quantitative comparison on the **RefCOCO** (val) to verify this.\n\n| Order      | PointM | mIoU |\n| :-: |:-: | :-: |\n| q1, q2, q3 | 60.0   | 31.3 |\n| q1, q3, q2 | 59.7   | 31.2 |\n\nReferences:\n\n[A] Denseclip: Language-guided dense prediction with context-aware prompting CVPR2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xsrcsqrO5c", "reviewer": "Reviewer_Tfcw", "summary": "This paper proposes the Progressive Comprehension Network (PCNet) for the WRIS task. this model achieves visual localization by progressively incorporating target-related textual cues for visual-linguistic alignment. Although experimental results have demonstrated the effectiveness of this paper, several aspects of the paper lack clarity.", "questions": "None", "limitations": "See the weakness", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe corresponding experiments validate the effectiveness of the proposed method on three popular benchmarks. the proposed method outperforms existing methods.\n2.\tThe two loss functions introduced in this paper effectively enhance the accuracy of the response maps.", "weaknesses": "1.\tThe Conditional Referring Module (CRM) is implemented through multiple cross-attentions, which is a relatively common approach, lacking novelty. \n2.\tWhat is the general idea of classification loss mentioned in the paper? Please explain the fundamental research insights of this work.\n3.\tIn the Region-aware Shrinking (RaS) loss, what does the ⊙ symbol represent in Equation 5? Please clarify its meaning. \n4.\tThis article only conducted ablation experiments on the RefCOCOg dataset, so the ablation experiments are insufficient. It is recommended to validate on multiple datasets (such as RefCOCO+, RefCOCO) to comprehensively evaluate the effectiveness of our method.\n5.\tThe order of the tables is somewhat confusing, with Table 9 preceding Table 6. It is recommended to reorder the tables to make them more in line with logical sequence or reading habits.\n6.\tWhich datasets were the ablation experiments in Tables 7-9 conducted on? Please specify.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincerely thanks for useful comments. To the weaknesses, our response is as follows:\n\n>**[W1]**: Conditional Referring Module (CRM) is implemented through multiple cross-attentions, which is a relatively common approach, lacking novelty.\n\n Thanks for your comments. We would like to address the concerns about its novelty and highlight its importance in our approach:\n\n- Firstly, the proposed CRM module is well-motivated. In our method, we observe that referring text descriptions typically contain detailed cues on how to localize the target object.  By progressively integrating these  target-related textual cues from the input description, we hope to enhance the target localization step-by-step. Therefore, we introduced the CRM module to model this progressive comprehension process. More importantly, the module is combined with the `RaS` loss to implement instance-level supervision, which has not been explored in previous approaches.\n- The CRM module includes two cross-attentions, each playing a crucial role. The first, vision-to-text attention, aims to obtain vision-attended cue features by incorporating visual context information. The second, text-to-text attention,  modulates the referring query using the target-related textual cue embeddings.  Through the interaction between referring query and different textual cue embeddings, this module is designed to achieve a more discriminative referring embedding. In Table 4, we provide ablation experiments for the module, which also validate the effectiveness of our CRM design.\n\n\n>**[W2]**:  the general idea of classification loss mentioned in the paper.\n\nWe sincerely apologize for the less detailed presentation about the Classification loss  $\\mathcal{L}\\_{\\texttt{cls}}$ in **TRIS** [25] due to space restriction. The loss $\\mathcal{L}\\_{\\texttt{cls}}$ aims to establish a global alignment between the visual content and the referring texts by formulating a classification process. We provided a detailed explanation of the idea and implementation of the loss in the **general response**. We will include this in the revision.\n\n>**[W3]**: what does the ⊙ symbol represent in Equation 5.\n\n**Ans**: The $\\odot$ denotes the hadamard product. In Equation 5, we multiply the binary mask obtained from SAM and the response map element by element. Thanks for your comments. We will add the explanation in the updated version.\n\n>**[W4]**: this article only conducted ablation experiments on the RefCOCOg dataset, so the ablation experiments are insufficient.\n\nIn our main paper, we follow  previous works, e.g., SAG[14] and TRIS[25], to conduct the ablation experiments on the  representative RefCOCOg dataset.  As suggested,  we conducted more ablations on other datasets (i.e., refcoco, refcoco+ datasets)  and shown the results in the following tables. The results on these datasets lead to the same conclusions as those on RefCOCOg, further validating the necessity and effectiveness of each component of our method.\n\n- RefCOCO\n\n|                              |                  |                              |                              |                 |                 |\n| :---------- | :--------- | ----------- | -------- | ------ | ------ |\n| $\\mathcal{L}_{\\texttt{CLs}}$ | ${\\texttt{CRM}}$ | $\\mathcal{L}_{\\texttt{RaS}}$ | $\\mathcal{L}_{\\texttt{IaD}}$ | **PointM**      | **mIoU**        |\n| &#10003;                     |                  |                              |                              | 50.3            | 24.6            |\n| &#10003;                     | &#10003;         |                              |                              | 53.5            | 26.4            |\n| &#10003;                     | &#10003;         | &#10003;                     |                              | 56.1            | 28.7            |\n| &#10003;                     | &#10003;         |                              | &#10003;                     | 56.8            | 28.0            |\n| &#10003;                     | &#10003;         | &#10003;                     | &#10003;                     | $\\textbf{60.0}$ | $\\textbf{31.3}$ |\n\n- RefCOCO+\n\n| $\\mathcal{L}_{\\texttt{CLs}}$ | ${\\texttt{CRM}}$ | $\\mathcal{L}_{\\texttt{RaS}}$ | $\\mathcal{L}_{\\texttt{IaD}}$ | PointM          | mIoU            |\n| :--------- | :-------- | ------- | ---------- | -------- | --------- |\n| &#10003;                     |                  |                              |                              | 44.5            | 22.1            |\n| &#10003;                     | &#10003;         |                              |                              | 48.8            | 24.6            |\n| &#10003;                     | &#10003;         | &#10003;                     |                              | 55.3            | 27.1            |\n| &#10003;                     | &#10003;         |                              | &#10003;                     | 52.4            | 25.9            |\n| &#10003;                     | &#10003;         | &#10003;                     | &#10003;                     | $\\textbf{58.7}$ | $\\textbf{29.2}$ |\n\n>**[W5]**: The order of the tables is somewhat confusing, with Table 9 preceding Table 6.....\n\nThanks for pointing out the problem. We will reorder the two tables to make them more in line with logical sequence or reading habits in the updated version.\n\n>**[W6]**: Which datasets were the ablation experiments in Tables 7-9 conducted on? Please specify.\n\nWe conduct the ablation experiments on **Refcocog** (google split) dataset as described in `lines 275-276`.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CIoD3wm9ak", "reviewer": "Reviewer_K7SG", "summary": "Aiming for the WRIS task, the work proposes a novel framework to leverage target-related textual cues from the description for progressively localizing the target object. The authors use a Large Language Model (LLM) to decompose the input text description into short phrases which are taken as target-related cues and fed into a Conditional Referring module. Besides, Region-aware Shrinking (RaS) loss and Instance-aware Disambiguation (IaD) loss are proposed to facilitate fine-grained cross-modal alignment.", "questions": "1. The mask proposals by SAM are often part of the whole instance.  It will affect the refined results. Does this work do anything special to alleviate the problem? The author can give more details about this issue.\n\n2. Table 1 shows the results of the pseudo labels obtained at the wealy-supervised training stage. Can the authors give the results after pseudo labels based on supervised training?", "limitations": "NaN", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The motivation is clear and easy to understand. The idea of leveraging target-related textual cues decomposed by LLM for progressive localization makes sense.\n\n2. This work develops multiple consecutive Conditional Referring Modules. RaS and IaD two loss objectives are seamlessly integrated into the framework.\n\n3.  It is novel to implement RaS and IaD two loss objectives by harnessing the capabilities of segmentation foundation models. The SAM[1] does not include strong semantic prior knowledge like GroundingDINO and Grounded-SAM[2], and it is suitable for the weakly-supervised setting.\n\n4. The work shows outstanding object localization ability and outperforms counterparts on three common benchmarks. Especially after refined by the SAM, the results are promising.\n\n[1] Segment anything ICCV2023\n\n[2] Grounding dino: Marrying dino with grounded pre-training for open-set object detection", "weaknesses": "1. In table 1, only the results refined by SAM[1] are reported. Considering that FreeSOLO [2] is an unsupervised segmentation model, I am curious about the results refined by FreeSOLO. \n\n2.  This work introduces multiple stages for progressive localization. It may increase the training and inference time of the model.  The time comparison of model training and inference may be needed.\n\n[1] Segment anything ICCV2023\n\n[2] Freesolo: Learning to segment objects without annotations CVPR2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **[W1]**: In table 1, only the results refined by SAM[16] are reported. Considering that FreeSOLO [40] is an unsupervised segmentation model, I am curious about the results refined by FreeSOLO.\n\n**[Ans]**: Thanks for your advice. Here we present the **mIoU** after applying FreeSOLO refinement, as shown in the table below. The first row displays the results for TRIS[25], while the second row presents the results for our proposed method. Although FreeSOLO's extracted mask proposals are not as accurate as those from SAM, leading to reduced performance, our method still exhibits clear and substantial improvements across three established benchmarks.\n\n| Method     | RefCOCO(val) | RefCOCO+ (val) | RefCOCOg_goolge(val) |\n| :-: | :-:|:-:|:-:|\n| TRIS[25] | 29.7         | 27.2           | 30.5                 |\n| PCNet      | **33.1**     | **30.3**       | **33.8**             |\n\n\n\n> **[W2]**: This work introduces multiple stages for progressive localization. It may increase the training and inference time of the model. The time comparison of model training and inference may be needed.\n\n**[Ans]**: Thanks for your comment. Our method incorporates a multi-stage refinement process, which naturally leads to increased training and inference times. Below, we provide a comparative analysis of the time costs associated with various methods. Notably, the overall time expenditure of our method with three stages remains well within acceptable limits.\n\n| Methods    | Training-Time | Inferring time |\n| :-: |  :-: |  :-: |\n| SAG[14]  | 36.0 h        | 63 ms          |\n| TRIS[25] | 3.0 h         | 35 ms          |\n| PCNet      | 6.0 h         | 42 ms          |\n\n\n\n\n\n> **[Q1]**: The mask proposals by SAM are often part of the whole instance. It will affect the refined results. Does this work do anything special to alleviate the problem? The author can give more details about this issue.\n\n**[Ans]**: You are correct that SAM's mask proposals often represent only a portion of the complete instance. Consequently, even when our method achieves accurate localization, it may not necessarily translate to precise instance masks. To demonstrate this, the last row (\"Oracle\") of Table 1 presents the mIoU scores achieved by selecting the mask proposal that best aligns with the ground-truth mask. These results support our observation and suggest that integrating a more advanced instance detector, such as GroundSAM [A], could further enhance our method's mask prediction accuracy (mIoU).\n\n> **[Q2]**: Table 1 shows the results of the pseudo labels obtained at the weakly-supervised training stage. Can the authors give the results after pseudo labels based on supervised training?\n\n**[Ans]**: Thanks for your question. We present additional results below. In our main paper, we focused on the quality of pseudo masks generated by our method (i.e., one stage). Here, we further investigate their effectiveness by training the LAVT [44] network using these pseudo masks as supervision. We report the resulting PointM (first row) and mIoU  (second row) metrics on three benchmarks, along with the performance achieved when refining mask predictions with SAM (third row). These findings highlight the potential for further improvement in mask prediction through fully supervised training based on our generated pseudo masks. We will incorporate these results into the revised version of our paper.\n\n| Metric                    | RefCOCO(val) | RefCOCO+ (val) | RefCOCOg_goolge(val) |\n| :-: | :-: | :-: | :-: | \n| **PointM**                | 66.1         | 60.4           | 62.4                 |\n| **mIoU**                  | 44.6         | 40.3           | 41.1                 |\n| **mIoU** (refined by SAM) | **54.5**     | **49.9**       | **51.1**             |\n\nReferences:\n\n[A] Grounding dino: Marrying dino with grounded pre-training for open-set object detection ECCV2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**To Reviewers and AC:** 

We extend our sincere gratitude to all the reviewers (**R1**-oLx4, **R2**-kTSb, **R3**-Tfcw, and **R4**-K7SG) for your time and insightful reviews, which help us emphasize the contributions of our work and revise the presentation. We are encouraged to hear that the reviewers found the work is well-motivated with good presentation and contribution (**R1**, **R4**), as well as the comprehensive experimental evaluation and commendable performance (**R1**, **R2**, **R3**, **R4**). We have methodically addressed each point in our individual responses, hoping that we can address your concerns. 

Here we first address broader questions:

**Explanation about the idea and implementation of classification loss $\mathcal{L}\_{\texttt{cls}}$ in TRIS** [25]: 

- Our work consists of multiple stages and utilizes $\mathcal{L}\_{\texttt{cls}}$ in TRIS at each stage independently for response maps optimization. Here, we omit the index of stage n for clarity. 

- $\mathcal{L}\_{\texttt{cls}}$ formulates the target localization problem as a classification process to differentiate between positive and negative text expressions. While the referring text expressions for an image are used as positive expressions, the referring text expressions from other images can be used as negative expressions for this image.  Thus, given a batch (i.e., B) of image samples , each image sample is mutually associated with one positive reference text (i.e., a text describing a specific object in the current image) and mutually exclusive with $L$ negative reference texts (texts that are not related to the target object in the image). Note that the number of batches is equal to the sum of the positive samples and the negative samples (i.e., $B = 1 + L$). 

- Speficially, in each training batch,  $B$ (i.e., $1+L$) image-text pairs $\\{ \mathbf{I}\_i, \mathbf{T}\_i \\}\_{i=1}^{B}$ are sampled. Through the language and vision encoders, we can get  referring embeddings ${\mathbf{Q}} \in \mathbb{R}^{B \times C}$ and image embeddings ${\mathbf{V}} \in \mathbb{R}^{B \times H \times W \times C}$. Then, we obtain the response maps ${\mathbf{R}} \in \mathbb{R}^{B \times B \times H \times W}$  by applying similarity calculation and normalization operation. After the pooling operation as done in TRIS, we further obtain the alignment score matrix ${\mathbf{y}} \in {\mathbb{R}}^{B \times B}$.  According to the $\mathcal{L}\_{\texttt{cls}}$, for $i\_{th}$ image in the batch, there is a prediction score $\mathbf{y}{[i, :]}$, where $\mathbf{y}{[i, i]}$ predicted by the corresponding text deserves a higher value (i.e, **1** positive one) and the others deserve lower values (**L** negative ones). 

  Then `Classification` loss for the $i\_{th}$ image from the batch can be formulated as cross-entropy loss:
 $$
\mathcal{L}\_{\texttt{cls}, i} = - \frac{1}{B} \sum\_{j=1}^{B}\left( \mathbb{1}\_{i=j} \log\left( \frac{1}{1+e^{-\mathbf{y}{[i, j]}}} \right) + (1-\mathbb{1}\_{i=j}) \log \left(\frac{e^{-\mathbf{y}{[i, j]}}}{1+e^{-\mathbf{y}{[i, j]}}} \right) \right),
 $$
​        and the `Classification` loss for the batch can be formulated as:
$$
\mathcal{L}\_{\texttt{cls}}= \frac{1}{B} \sum\_{i=1}^{B} \mathcal{L}\_{\texttt{cls}, i}
$$
​        The $i$ denotes the index for the visual image and the $j$ denotes the index for the text.



We address the raised concerns below and will revise our paper according to all comments. Please let us know if you have further questions.

Regards,

Authors (paper 891)

Meta-review: Initially, the paper received mixed reviews, but following the author's rebuttal, the reviewers have reached a positive consensus. Each reviewer noted that the authors effectively addressed all concerns raised in their initial feedback. Two reviewers now strongly support the paper with a score of 7. The paper's contributions are recognized for their clear motivation, strong presentation, originality, and extensive experimental support. The AC also agrees that the paper exceeds the acceptance standards.
Decision: Accept (poster)