Title: Boosting Weakly Supervised Referring Image Segmentation via Progressive Comprehension
Venue: NeurIPS 2024
Accepted: yes
Keywords: multi-modal

segmentation

weakly-supervised
Abstract: This paper explores the weakly-supervised referring image segmentation (WRIS) problem, and focuses on a challenging setup where target localization is learned directly from image-text pairs. 
We note that the input text description typically already contains detailed information on how to localize the target object, and we also observe that humans often follow a step-by-step comprehension process (\ie, progressively utilizing target-related attributes and relations as cues) to identify the target object. 
Hence, we propose a novel Progressive Comprehension Network (PCNet) to leverage target-related textual cues from the input description for progressively localizing the target object.
Specifically, we first use a Large Language Model (LLM) to decompose the input text description into short phrases. These short phrases are taken as target-related cues and fed into a Conditional Referring Module (CRM) in multiple stages, to allow updating the referring text embedding and enhance the response map for target localization in a multi-stage manner.
Based on the CRM, we then propose a Region-aware Shrinking (RaS) loss to constrain the visual localization to be conducted progressively in a coarse-to-fine manner across different stages.
Finally, we introduce an Instance-aware Disambiguation (IaD) loss to suppress instance localization ambiguity by differentiating overlapping response maps generated by different referring texts on the same image. 
Extensive experiments show that our method outperforms SOTA methods on three common benchmarks.

Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehension

Abstract

This paper explores the weakly-supervised referring image segmentation (WRIS) problem, and focuses on a challenging setup where target localization is learned directly from image-text pairs. We note that the input text description typically already contains detailed information on how to localize the target object, and we also observe that humans often follow a step-by-step comprehension process ( i.e. , progressively utilizing target-related attributes and relations as cues) to identify the target object. Hence, we propose a novel Progressive Comprehension Network (PCNet) to leverage target-related textual cues from the input description for progressively localizing the target object. Specifically, we first use a Large Language Model (LLM) to decompose the input text description into short phrases. These short phrases are taken as target-related cues and fed into a Conditional Referring Module (CRM) in multiple stages, to allow updating the referring text embedding and enhance the response map for target localization in a multi-stage manner. Based on the CRM, we then propose a Region-aware Shrinking (RaS) loss to constrain the visual localization to be conducted progressively in a coarse-to-fine manner across different stages. Finally, we introduce an Instance-aware Disambiguation (IaD) loss to suppress instance localization ambiguity by differentiating overlapping response maps generated by different referring texts on the same image. Extensive experiments show that our method outperforms SOTA methods on three common benchmarks.

1 Introduction

Referring Image Segmentation (RIS) aims to segment a target object in an image via a user-specified input text description. RIS has various applications, such as text-based image editing [17, 13, 2] and human-computer interaction [62, 51]. Despite remarkable progress, most existing RIS works [7, 58, 27, 26, 21, 5] rely heavily on pixel-level ground-truth masks to learn visual-linguistic alignment. Recently, there has been a surge in interest in developing weakly-supervised RIS (WRIS) methods via weak supervisions, e.g. , bounding-boxes [9], and text descriptions [54, 18, 30, 4], to alleviate burden of data annotations. In this work, we focus on obtaining supervision from text descriptions only.

The relatively weak constraint of utilizing text alone as supervision makes visual-linguistic alignment particularly challenging. There are some attempts [30, 18, 22, 46] to explore various alignment workflows. For example, TRIS [30] classifies referring texts that describe the target object as positive texts while other texts as negative ones, to model a text-to-image response map for locating potential target objects. SAG [18] introduces a bottom-up and top-down attention framework to discover

(a) Image

Q: '

a player wearing a blue and

gray uniform catches a ball'

(e) TRIS

(d) GT (Step-3)

(f) Ours (Step-1)

(g) Ours (Step-2)

(h) Ours (Step-3)

Q3:

'catches a ball'

(b) GT (Step-1)

Q1:

'a player'

+

(c) GT (Step-2)

Q2:

'blue and gray uniform'

+

individual entities and then combine these entities as the target of the referring expression. However, these methods encode the entire referring text as a single language embedding. They can easily overlook some critical cues related to the target object in the text description, leading to localization ambiguity and even errors. For example, in Fig. 1(e), TRIS [30] erroneously activates all three players due to its use of cross-modality interactions between the image and the complete language embedding only.

We observe that humans typically localize target objects through a step-by-step comprehension process. Cognitive neuroscience studies [48, 41] also support this observation, indicating that humans tend to simplify a complex problem by breaking it down into manageable sub-problems and reasoning them progressively. For example, in Fig. 1(b-d), human perception would first begin with 'a player' and identify all three players (b). The focus is then refined by the additional detail 'blue and gray uniform' , which helps exclude the white player on the left (c). Finally, the action 'catches a ball' helps further exclude the person on the right, leaving the correct target person in the middle (d).

Inspired by the human comprehension process, we propose in this paper a novel Progressive Comprehension Network (PCNet) for WRIS. We first employ a Large Language Model (LLM) [59] to dissect the input text description into multiple short phrases. These decomposed phrases are considered as target-related cues and fed into a novel Conditional Referring Module (CRM), which helps update the global referring embedding and enhance target localization in a multi-stage manner. We also propose a novel Region-aware Shrinking (RaS) loss to facilitate visual localization across different stages at the region level. ReS first separates the target-related response map (indicating the foreground region) from the target-irrelevant response map (indicating the background region), and then constrains the background response map to progressively attenuate, thus enhancing the localization accuracy of the foreground region. Finally, we notice that salient objects in an image can sometimes trigger incorrect response map activation for text descriptions that aim for other target objects. Hence, we introduce an Instance-aware Disambiguation (IaD) loss to reduce the overlapping of the response maps by rectifying the alignment score of different referring texts to the same object.

In summary, our main contributions are as follows :

We propose the Progressive Comprehension Network (PCNet) for the WRIS task. Inspired by the human comprehension processes, this model achieves visual localization by progressively incorporating target-related textual cues for visual-linguistic alignment.

Our method has three main technical novelties. First, we propose a Conditional Referring Module (CRM) to model the response maps through multiple stages for localization. Second, we propose a Region-aware Shrinking (RaS) loss to constrain the response maps across different stages for better cross-modal alignment. Third, to rectify overlapping localizations, we propose an Instance-aware Disambiguation (IaD) loss for different referring texts paired with the same image.

We conduct extensive experiments on three popular benchmarks, demonstrating that our method outperforms existing methods by a remarkable margin.

2 Related work

Referring Image Segmentation (RIS) aims to segment the target object from the input image according to the input natural language expression. Hu et al. [14] proposes the first CNN-based RIS method. There are many follow-up works. Early methods [60, 28, 38] focus on object-level cross-modal alignment between the visual region and the corresponding referring expression. Later, many works explore the use of attention mechanisms [15, 7, 58, 19] or transformer architectures [58, 29] to model long-range dependencies, which can facilitate pixel-level cross-model alignment. For example, CMPC [15] employs a two-stage progressive comprehension model to first perceive all relevant instances through entity wording and then use relational wording to highlight the referent. In contrast, our approach leverages LLMs to decompose text descriptions into short phrases related to the target object, focusing on sentence-level (rather than word-level) comprehension, which aligns more closely with human cognition. Focusing on the visual grounding, DGA [56] also adopts multi-stage refinement. It aims to model visual reasoning on top of the relationships among the objects in the image. Differently, our work addresses the weakly RIS task and aims to alleviate the localization ambiguity by progressively integrating fine-grained attribute cues.

Weakly-supervised RIS (WRIS) has recently begun to attract some attention, as it can substantially reduce the burden of data labeling especially on the segmentation field [25, 57, 63]. Feng et al. [9] proposes the first WRIS method, which uses bounding boxes for annotations. Several subsequent works [18, 22, 30] attempt to use weaker supervision signal, i.e. , text descriptions. SAG [18] proposes to first divide image features into individual entities via bottom-up attention and then employ top-down attention to learn relations for combining entities. Lee et al. [22] generate Grad-CAM for each word of the description and then consider the relations using an intra-chunk and inter-chunk consistency. Instead of merging individual responses, TRIS [30] directly learns the text-to-image response map by contrasting target-related positive and target-unrelated negative texts. Inspired by the generalization capabilities of segmentation foundation models [20, 8, 34], PPT [6] enables effective integration with pre-trained language-image models [43, 33] and SAM [20] by a lightweight point generator to identify the referent and context noise. Despite their success, these methods encode the full text as a single embedding for cross-modality alignment, which overlooks target-related nuances in the textual descriptions. In contrast, our method proposes to combine progressive text comprehension and object-centric visual localization to obtain better fine-grained cross-modal alignment.

Large Language Models (LLMs) are revolutionizing various visual domains, benefited by their userfriendly interfaces and strong zero-shot prompting capabilities [3, 49, 1, 47]. Building on this trend, recent works [42, 55, 53, 45, 64] explore the integration of LLMs into vision tasks ( e.g. , languageguided segmentation [55, 53], relation [23], and image classification [42]) through parameter-efficient fine-tuning or knowledge extraction. For example, LISA [55] and GSVA [53] utilize LLaVA [32], a large vision-language model (LVLM), as a feature encoder to extract visual-linguistic crossmodality features and introduce a small set of trainable parameters to prompt SAM [20] for reasoning segmentation. RECODE [23] and CuPL [42] leverage the knowledge in LLMs to generate informative descriptions as prompts for different categories classification. Unlike these works, we capitalize on the prompt capability of LLMs to help decompose a single referring description into multiple target object-related phrases, which are then used in our progressive comprehension process for RIS.

3 Our Method

In this work, we observe that when identifying an object based on a description, humans tend to first pinpoint multiple relevant objects and then narrow their focus to the target through step-by-step reasoning [48, 41]. Inspired by this, we propose a Progressive Comprehension Network (PCNet) for WRIS, which enhances cross-modality alignment by progressively integrating target-related text cues at multiple stages. Fig. 2 shows the overall framework of our PCNet.

Given an image I and a referring expression T as input, we first feed T into a Large Language Model (LLM) to break it down into K short phrases T sub = { t 0 , t 1 , · · · , t K -1 } , referred to as target-related text cues. We then feed image I and referring expression T and the set of short phrases T sub into image encoder and text encoder to obtain visual feature V 0 ∈ R H × W × C v and language

LLM

Referring

Modulation

Cls

RaS

Stage 0

Cls

Referring

Modulation

Cls

RaS

In context instruction

…

Cls

Classification loss

Image-Text

Pairs

IaD

Instance-aware Disambiguation loss

CRM

Conditional Referring Module

Concatenation

IaD

'a player wearing a blue and

gray uniform catches a ball'

…

RaS

Region-aware Shrinking loss

Text clues:

1. A player

2. He wears blue/gray uniform

3. He catches a ball

IaD

IaD

Stage 1

Stage N-1

Response Map

Generation

Response Map

Generation

Response Map

Generation

Referring

Modulation

…

…

…

…

…

Proposal

Generator

Image

Encoder

Text

Encoder

I

T

Ro,a

Ro,d

R1,a

R1,d

RN-1,a

RN-1,d

Ro

R

RN-1

Qi1Z

20

QNV

VN-1

Vo

V1

Tsub

Qo_

90

Q1_

1

IN-1

Qsub

feature Q 0 ∈ R 1 × C t , and Q sub = { q 0 , q 1 , · · · , q K -1 } , with q k ∈ R 1 × C t , where H = H I /s and W = W I /s . C v and C l denote the numbers of channels of visual and text features. s is the ratio of down-sampling. We then use projector layers to transform the visual feature V 0 and textual features Q 0 and Q sub to a unified dimension C , i.e. , V 0 ∈ R H × W × C , Q 0 and q i are in R 1 × C .

We design PCNet with multiple consecutive Conditional Referring Modules (CRMs) to progressively locate the target object across N stages 1 . Specifically, at stage n , the n -th CRM updates the referring embedding Q n into Q n +1 conditioned on the short phrase q n from the proposed Referring Modulation block. Both Q n +1 and visual embedding V n are fed into Response Map Generation to generate the text-to-image response map R n and updated visual embedding V n +1 . Finally, the response map R N -1 generated by the n -th CRM is used as the final localization result. To optimize the resulting response map for accurate visual localization, we employ the pre-trained proposal generator to obtain localization-matched mask proposals. We also propose Region-aware Shrinking (RaS) loss to constrain the visual localization in a coarse-to-fine manner, and Instance-aware Disambiguation (IaD) loss to suppress instance localization ambiguity.

In the following subsections, we first discuss how we decompose the input referring expression into target-related cues in Sec. 3.1. We then introduce the CRM in Sec. 3.2. Finally, we present our Region-aware Shrinking loss in Sec. 3.3, and Instance-aware Disambiguation loss in Sec. 3.4.

3.1 Generation of Target-related Cues

Existing works typically encode the entire input referring text description, and can easily overlook some critical cues ( e.g. , attributes and relations) in the description (particularly for a long/complex description), leading to target localization problems. To address this problem, we propose dividing the input description into short phrases to process it individually. To do this, we leverage the strong in-context capability of the LLM [1] to decompose the text description. We design a prompt, with four parts, to instruct the LLM to do this: (1) general instruction P G ; (2) output constraints P C ; (3) in-context task examples P E ; and (4) input question P Q . P G describes the overall instruction, e.g. 'decomposing the referring text into target object-related short phrases'. P C elaborates the output setting, e.g. , sentence length of each short phrase. In P E , we specifically curate several in-context pairs as guidance for the LLM to generate analogous outputs. Finally, P Q encapsulates the input text description and the instruction words for the LLM to execute the operation. The process of generating

target-related cues is formulated as:

where K represents the total number of phrases, which varies depending on the input description. Typically, longer descriptions more likely yield more phrases. To maintain consistency in our training dataset, we standardize it to five phrases ( i.e. , K = 5 ). If fewer than five phrases are produced, we simply duplicate some of the short phrases to obtain five short phrases. In this way, phrases generated by LLM are related to the target object and align closely with our objective.

3.2 Conditional Referring Module (CRM)

Given the decomposed phrases ( i.e. , target-related cues), we propose a CRM to enhance the discriminative ability on the target object region conditioned on these phrases, thereby improving localization accuracy. As shown in Fig. 2, the CRM operates across N consecutive stages. At each stage, it first utilizes a different target-related cue to modulate the global referring embedding via a referring modulation block and then produces the image-to-text response map through a response map generation block.

Referring Modulation Block. Considering the situation at stage n , we first concatenate one targetrelated cue q n and the L negative text cues obtained from other images 2 , to form q ′ n ∈ R ( L +1) × C . We then fuse the visual features V n with q ′ n through a vision-to-text cross-attention, to obtain vision-attended cue features ˆ q n ∈ R ( L +1) × C , as:

where A v → t ∈ R ( L +1) × H × W denotes the vision-to-text inter-modality attention weight. W V ∗ and W q ′ ∗ are learnable projection layers. ⊗ denotes matrix multiplication. Using the vision-attended cue features ˆ q n , we then enrich the global textual features Q n into cue-enhanced textual features Q n +1 ∈ R 1 × C through another text-to-text cross-attention, as:

where A t → t ∈ R 1 × ( L +1) represents the text-to-text intra-modality attention weight. W Q ∗ and W ˆ q ∗ are learnable projection layers. In this way, we can enhance the attention of Q n on the target object by conditioning its own target-related cue features and the global visual features.

Response Map Generation. To compute the response map, we first update the visual features V n to ˆ V n by integrating them with the updated referring text embedding Q n +1 using a text-to-visual cross-attention, thereby reducing the cross-modality discrepancy. Note that ˆ V n is then used in the next stage ( i.e. , V n +1 = ˆ V n ). The response map R n ∈ R H × W at the n -th stage is computed as:

where Norm normalizes the output in the range of [0 , 1] . To achieve global visual-linguistic alignment, we adopt classification loss L Cls in [30] to optimize the generation of the response map at each stage. It formulates the target localization problem as a classification process to differentiate between positive and negative text expressions. While the referring text expressions for an image are used as positive expressions, the ones from other images can be used as negative for this image. More explanations are given in appendix.

3.3 Region-aware Shrinking (RaS) Loss

Despite modulating the referring attention with the target-related cues stage-by-stage, image-text classification often activates irrelevant background objects due to its reliance on global and coarse response map constraints. Ideally, as the number of target-related cues used increases across each stage, the response map should become more compact and accurate. However, directly constraining the latter stage to have a more compact spatial activation than the former stage can lead to a trivial

solution ( i.e. , without target activation). To address this problem, we propose a novel region-aware shrinking (RaS) loss, which segments the response map into foreground (target) and background (non-target) regions. Through contrastive enhancement between these regions, our method gradually reduces the background interference while refining the foreground activation in the response map.

Specifically, at stage n , we first employ a pretrained proposal generator to obtain a set of mask proposals, M = { m 1 , m 2 , · · · , m P } , where each proposal m p is in R H × W and P is the total number of segment proposals. We then compute a alignment score between the response map R n and each proposal m p in M as:

where ⊙ denotes the hadamard product. The proposal with the highest score (denoted as m f ) is then treated as the target foreground region, while the combination of other proposals (denoted as m b ) is regarded as non-target background regions. With the separated regions, we define a localization ambiguity S amb n , which measures the uncertainty of the target object localization in the current stage n , as:

where S amb n is in the range of [0 , 1] , and IoU denotes the intersection over union. When the localization result ( i.e. , the response map) matches the only target object proposal instance exactly, ambiguity is 0. Conversely, if it matches the more background proposals, ambiguity approaches 1.

Assuming that each target in the image corresponds to an instance, by integrating more cues, the model will produce a more compact response map and gradually reduce the ambiguity. Consequently, based on the visual localization results from two consecutive stages, we can formulate the region-aware shrinking objective for a total of N stages as:

By introducing region-wise ambiguity, L RaS can direct non-target regions to converge towards attenuation while maintaining and improving the quality of the response map in the target region. This enables the efficient integration of target-related textual cues for progressively finer cross-modal alignment. Additionally, the mask proposals can also provide a shape prior to the target region, which helps to further enhance the accuracy of the target object localization.

3.4 Instance-aware Disambiguation (IaD) Loss

Although the RaS loss can help improve the localization accuracy by reducing region-wise ambiguity within one single response map, it takes less consideration of the relation between different instancewise response maps. Particularly, we note that, given different referring descriptions that refer to different objects of an image, there are usually some overlaps among the corresponding response maps. For example, in Fig. 2, the player in the middle is simultaneously activated by two referring expressions ( i.e. , the response maps R ∗ , a and R ∗ , d have overlapping activated regions), resulting in inaccurate localization. To address this problem, we propose an Instance-aware Disambiguation (IaD) loss to help enforce that different regions of the response maps within a stage are activated if the referring descriptions of an image refer to different objects.

Specifically, given a pair of image I a and input text description T a , we first sample extra N d text descriptions, T d = { t 1 , t 2 , · · · , t N d } , where the referred target object of each text description t d is in the image I a but is different from the target object referred to by T a . We then obtain the image-to-text response maps R a and R d = { R 1 , R 2 , · · · , R N d } for T a and T d through Eq. (4). Here, we omit the stage index n for clarity. Then, based on the Eq. (5), we obtain the alignment scores S a and {S d } N d d =1 for T a and T d . In S , the larger the value, the higher the alignment between the corresponding proposal (specified by the index) and the current text. To disambiguate overlapping activated regions, we constrain that the maximum index of the alignment score between S a and each of S d must be different from each other ( i.e. , different texts must activate different objects). Here, we follow [50] to compute the index vector, y ∈ R 1 × P , as:

where sg ( · ) represents the stop gradient operation. Finally, we denote the index vectors for S a and {S d } N d d =1 as y a and { y d } N d d =1 , and we formulate the IaD loss as:

By enforcing the constraint at each stage, the response maps activated by different referring descriptions in an image for different instances are separated, and the comprehension of the discriminative cues is further enhanced.

4 Experiments

4.1 Settings

Dataset. We have conducted experiments on three standard benchmarks: RefCOCO [61], RefCOCO+ [61], and RefCOCOg [39]. They are constructed based on MSCOCO [24]. Specially, the referring expressions in RefCOCO and RefCOCO+ focus more on object positions and appearances, respectively, and they are characterized by succinct descriptions, averaging 3.5 words in length. RefCOCOg contains much longer sentences (average length of 8.4 words), making it more challenging than others. RefCOCOg includes two partitions: UMD [40] and Google [39].

Implementation Details. We train our framwork for 15 epochs with a batch size of 36 on an RTX4090 GPU. The total loss for training is L total = L Cls + L RaS + L IaD . By default, we set the number of stages N to 3, and the number of the additional text descriptions sampled for each image N d to 1. Without loss of generality, we use FreeSOLO [52] and SAM [20] as the proposal generators to obtain two versions: PCNet S and PCNet F . Refer to Sec. A for more implementation details.

c)

'the zebra that is behind the front zebra and the elephant'

'zebra looking at the camera through the legs of an elephant'

'a child sitting on a chair

with a blue shirt on'

'a woman wearing a green

shirt and black pants'

'base ball player ready to

swing'

'man squatting in a head gear

between the batter and umpire'

'a nab wearing a

white shirt and tie'

'a white - haired man

talking on a phone'

Referring Expressions:

Referring Expressions:

Referring Expressions:

(a)

(d)

(b)

(c)

'a sheep eating grass'

Referring Expressions:

'a small sheep standing

between two other sheep'

Q

: '3 teddy bears sitting on a bed'

(a) Image (b) GT (c) Ours Evaluation Metrics. We argue that the key to WRIS is target localization, and the performance evaluation should not rely primarily on pixel-wise metrics. With the accurate localization points, pixel-level masks can be readily obtained by prompting the pre-trained segmentors ( e.g. , SAM [20]). Thus, following [18, 22, 30], we adopt localization-based metric ( i.e. , PointM), and pixel-wise metrics ( i.e. , mean and overall intersection-over-union (mIoU and oIoU) for evaluation. PointM [30] is used to evaluate the localization accuracy, which computes the ratio of activation peaks in the mask region.

4.2 Comparison with State-of-the-Art Methods

Quantitative comparison. Tab. 1 compares our method with various SOTA methods. Specifically, we first compare target localization accuracy using the PointM metric. We evaluate two model variants: PCNet F and PCNet S , which use different segmentors ( e.g. , FreeSOLO [52] and SAM [20]) to extract mask proposals for RaS and IaD losses. Even when using FreeSOLO as the proposal generator, our model still significantly outperforms all compared methods. For example, on the most challenging dataset, RefCOCOg, with more complex object relationships and longer texts, PCNet F achieves performance improvements of 55.2% and 10.3% on the Val (G) set compared to SAG and TRIS 3 . PCNet S further boosts the performance if we replace FreeSOLO with the stronger SAM.

In addition, we verify the accuracy of the response map through pixel-wise mIoU metric. Results are shown in the middle part of Tab. 1. Our PCNet still achieves superior performances on all benchmarks, against all compared methods. Particularly, PCNet F and PCNet S outperform TRIS by an improvement of 10.8% and 14.1% mIoU, respectively, on the RefCOCOg Val (G) set. In the bottom part of Tab. 1, we compare the accuracy of the extracted mask proposals generated using the target localization point ( i.e. , the peak point of the response map) to prompt SAM. We can see that our PCNet significantly outperforms other WRIS methods. We can also see that higher PointM values correlate with higher mIoU accuracy values of the corresponding mask proposals for different methods. We further tested the mask accuracy using the Ground-Truth localization point ( i.e. , the last row), and find that its performance even surpasses the fully-supervised method, LA VT [58]. All these results highlight the critical importance of target localization ( i.e. , peak point) for the WRIS task.

In Fig. 3, we show some visual results of our method across different scenes by using the target localization point ( i.e. , the green marker) to prompt SAM to generate the target mask. Our method effectively localizes the target instance among other instances within the image, even in complex scenarios with region occlusion (a), multiple instances (b), similar appearance (c), and dim light (d).

4.3 Ablation Study

We conduct ablation experiments on the RefCOCOg dataset and report the results on the Val (G) set from both PCNet S and PCNet F in Tab. 2, and from PCNet F in Tab. 3, Tab. 4, and Tab. 5.

Component Analysis. In Tab. 2, we first construct a single-stage baseline ( 1st row) to optimize visual-linguistic alignment by removing all proposed components and using only the global imagetext classification loss L Cls . We then introduce the proposed conditional referring module (CRM) to the baseline to allow for multi-stage progressive comprehension ( 2nd row). To validate the efficacy of the region-aware shrinking loss (RaS) and instance-aware disambiguation loss (IaD), we introduce them separately ( 3rd and 4th rows). Finally, we combine all proposed components ( 5th row).

Referring Expressions:

The results demonstrate that ❶ even using only L Cls , progressively introducing target-related cues through CRM can still significantly enhance target object localization. In particular, PCNetS achieves improvements of 3.1% on PointM and 5.9% on mIoU; ❷ by using L RaS to constrain response maps, making them increasingly compact and complete during the progressive comprehension process, the accuracy of target localization is dramatically enhanced, resulting in an improvement of 11.6% on PointM. ❸ although L IaD can facilitate the separation of overlapping response maps between different instances within the same image and improve the discriminative ability of our model on the target object, the lack of constraints between consecutive stages results in a smaller performance improvement than L RaS; and ❹ all components are essential for our final PCNet, and combining them achieves the best performance. In Fig. 4, we also provide the visual results of the ablation study on two examples. We can see that each component can bring obvious localization improvement.

在此处键入公式

Number of Iterative Stages. In Tab. 3, we analyze the effect of the number of iterative stages N . When N = 1 , we can only apply L Cls and L IaD, but not L RaS , resulting in inferior results. Increasing N from 1 to 2 significantly improves the performance due to the progressive introduction of targetrelated cues. However, the improvement from N = 2 to N = 3 is less pronounced than from N = 1 to N = 2 , and the performance stabilizes at N = 3 . At N = 4 , the performance slightly declines. This is because when the effective short-phrases decomposed by LLM are fewer than the number of stages, we need to repeat text phrases in later stages, which may affect the loss optimization. Q: a boy in black t - shirt and jeans bending by keeping his hands on knees

Modulation Strategy. In Tab. 4, we ablate different variants of CRM: ❶ directly adding target cue features q n and global referring features Q n (denoted as ADD); ❷ fusing q n and Q n using only text-to-text cross-attention (denoted as TTA); ❸ first employing a vision-to-text cross-attention to fuse visual features V n and q n to obtain vision-attended features ˆ q n , and then adding them to Q n (denoted VTA+ADD). The results demonstrate that ADD is the least efficient method. TTA Q: a young man with a shirt that has a giant musical note on it Q: a white mixture with herbs that is spread on top of some dough

Q: '

a catcher rushing to make a play on the ball'

(a) Image

(f) GT

(b)

(c)

+

(d)

+

(e)

+

Q: '

a light brown color sweet vada with dark brown one next to it'

Q : '3 teddy bears sitting on a bed'

(a) Image

(b) GT

(c) Ours

口

outperforms ADD but is less effective than VTA+ADD, verifying the importance of the vision context. Finally, our CRM combines VTA and TTA and achieves the best results.

Number of Referring Texts. In Tab. 5, we analyze the effect of N d used in L LaD. The results show that N d = 1 is enough, and the performance deteriorates as N d increases. This is because an image typically has 2-3 text descriptions, which means N d should be 1-2. As N d increases, repeated sampling becomes more frequent, affecting model training and thus leading to poorer results.

man in white and red baseball uniform batting at plate

Q: 5 Conclusion

Q: man squatting in a protective head gear between the batter and umpire (a) Image (b) GT (c) wo/ IaD (d) w/ IaD In this paper, we have proposed a novel Progressive Comprehension Network (PCNet) to perfom progressive visual-linguistic alignment for the weakly-supervised referring image segmentation (WRIS) task. PCNet first leverages a LLM to decompose the input referring description into several target-related phrases, which are then used by the proposed Conditional Referring Module (CRM) to update the referring text embedding stage-by-stage, thus enhancing target localization. In addition, we proposed two loss functions, region-aware shrinking loss and instance-aware disambiguation loss, to facilitate comprehension of the target-related cues progressively. We have also conducted extensive experiments on three RIS benchmarks. Results show that the proposed PCNet achieves superior visual localization performances and outperforms existing SOTA WRIS methods by large margins.

Our method does have limitations. For example, as shown in Fig. 5, when the text description refers to multiple objects, our method fails to return all referring regions. This is because our model design always assumes that there is only one object referred to by the language expression. In the future, we plan to incorporate more fine-grained vision priors [35, 44] and open-world referring descriptions ( e.g. , camouflaged [12], semi-transparent [31], shadow [36] and etc. ) into the model design to enable a more generalized solution.

References

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv:2303.08774 (2023)

Avrahami, O., Lischinski, D., Fried, O.: Blended diffusion for text-driven editing of natural images. In: CVPR. pp. 18208-18218 (2022)

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. pp. 1877-1901 (2020)

Chen, D., Wu, Z., Liu, F., Yang, Z., Huang, Y., Bao, Y., Zhou, E.: Prototypical contrastive language image pretraining. arXiv preprint arXiv:2206.10996 (2022)

Chng, Y.X., Zheng, H., Han, Y., Qiu, X., Huang, G.: Mask grounding for referring image segmentation. arXiv:2312.12198 (2023)

Dai, Q., Yang, S.: Curriculum point prompting for weakly-supervised referring image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13711-13722 (2024)

Ding, H., Liu, C., Wang, S., Jiang, X.: Vision-language transformer and query generation for referring segmentation. In: ICCV. pp. 16321-16330 (2021)

Du, Y., Bai, F., Huang, T., Zhao, B.: Segvol: Universal and interactive volumetric medical image segmentation. arXiv preprint arXiv:2311.13385 (2023)

Feng, G., Zhang, L., Hu, Z., Lu, H.: Learning from box annotations for referring image segmentation. IEEE TNNLS (2022)

He, C., Li, K., Zhang, Y., Xu, G., Tang, L., Zhang, Y., Guo, Z., Li, X.: Weakly-supervised concealed object segmentation with sam-based pseudo labeling and multi-scale feature grouping. NeurIPS 36 (2024)

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770-778 (2016)

He, R., Dong, Q., Lin, J., Lau, R.W.: Weakly-supervised camouflaged object detection with scribble annotations. In: AAAI (2023)

Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-or, D.: Prompt-to-prompt image editing with cross-attention control. In: ICLR (2022)

Hu, R., Rohrbach, M., Darrell, T.: Segmentation from natural language expressions. In: ECCV. pp. 108-124 (2016)

Huang, S., Hui, T., Liu, S., Li, G., Wei, Y ., Han, J., Liu, L., Li, B.: Referring image segmentation via cross-modal progressive comprehension. In: CVPR. pp. 10488-10497 (2020)

Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.: Mistral 7b. arXiv preprint arXiv:2310.06825 (2023)

Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., Irani, M.: Imagic: Text-based real image editing with diffusion models. In: CVPR. pp. 6007-6017 (2023)

Kim, D., Kim, N., Lan, C., Kwak, S.: Shatter and gather: Learning referring image segmentation with text supervision. In: ICCV. pp. 15547-15557 (2023)

Kim, N., Kim, D., Lan, C., Zeng, W., Kwak, S.: Restr: Convolution-free referring image segmentation using transformers. In: CVPR. pp. 18145-18154 (2022)

Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: ICCV. pp. 4015-4026 (2023)

Lai, X., Tian, Z., Chen, Y., Li, Y ., Yuan, Y ., Liu, S., Jia, J.: Lisa: Reasoning segmentation via large language model. arXiv:2308.00692 (2023)

Lee, J., Lee, S., Nam, J., Yu, S., Do, J., Taghavi, T.: Weakly supervised referring image segmentation with intra-chunk and inter-chunk consistency. In: ICCV. pp. 21870-21881 (2023)

Li, L., Xiao, J., Chen, G., Shao, J., Zhuang, Y ., Chen, L.: Zero-shot visual relation detection via composite visual cues from large language models. In: NeurIPS (2024)

Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: ECCV. pp. 740-755 (2014)

Lin, Y., Chen, M., Wang, W., Wu, B., Li, K., Lin, B., Liu, H., He, X.: Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation. In: CVPR. pp. 15305-15314 (2023)

Liu, C., Ding, H., Jiang, X.: Gres: Generalized referring expression segmentation. In: CVPR. pp. 23592-23601 (2023)

Liu, C., Ding, H., Zhang, Y., Jiang, X.: Multi-modal mutual attention and iterative interaction for referring image segmentation. IEEE TIP (2023)

Liu, D., Zhang, H., Wu, F., Zha, Z.J.: Learning to assemble neural module tree networks for visual grounding. In: ICCV. pp. 4673-4682 (2019)

Liu, F., Kong, Y., Zhang, L., Feng, G., Yin, B.: Local-global coordination with transformers for referring image segmentation. Neurocomputing (2023)

Liu, F., Liu, Y., Kong, Y., Xu, K., Zhang, L., Yin, B., Hancke, G., Lau, R.: Referring image segmentation using text supervision. In: ICCV. pp. 22124-22134 (2023)

Liu, F., Liu, Y ., Lin, J., Xu, K., Lau, R.W.: Multi-view dynamic reflection prior for video glass surface detection. In: AAAI (2024)

Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2024)

Liu, Y., Wang, X., Zhu, M., Cao, Y., Huang, T., Shen, C.: Masked channel modeling for bootstrapping visual pre-training. International Journal of Computer Vision pp. 1-21 (2024)

Liu, Y., Zhu, M., Li, H., Chen, H., Wang, X., Shen, C.: Matcher: Segment anything with one shot using all-purpose feature matching. arXiv preprint arXiv:2305.13310 (2023)

Liu, Y., Ke, Z., Liu, F., Zhao, N., Lau, R.W.: Diff-plugin: Revitalizing details for diffusion-based low-level tasks. In: CVPR (2024)

Liu, Y., Ke, Z., Xu, K., Liu, F., Wang, Z., Lau, R.W.: Recasting regional lighting for shadow removal. In: AAAI (2024)

Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2018)

Luo, G., Zhou, Y., Sun, X., Cao, L., Wu, C., Deng, C., Ji, R.: Multi-task collaborative network for joint referring expression comprehension and segmentation. In: CVPR. pp. 10034-10043 (2020)

Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A.L., Murphy, K.: Generation and comprehension of unambiguous object descriptions. In: CVPR. pp. 11-20 (2016)

Nagaraja, V.K., Morariu, V.I., Davis, L.S.: Modeling context between objects for referring expression understanding. In: ECCV. pp. 792-807 (2016)

Plass, J.L., Moreno, R., Brünken, R.: Cognitive load theory. Cambridge University Press (2010)

Pratt, S., Covert, I., Liu, R., Farhadi, A.: What does a platypus look like? generating customized prompts for zero-shot image classification. In: ICCV. pp. 15691-15701 (2023)

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748-8763 (2021)

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022)

Salewski, L., Alaniz, S., Rio-Torto, I., Schulz, E., Akata, Z.: In-context impersonation reveals large language models' strengths and biases. In: NeurIPS (2024)

Shaharabany, T., Tewel, Y., Wolf, L.: What is where by looking: Weakly-supervised open-world phrase-grounding without text inputs. pp. 28222-28237 (2022)

Shao, H., Qian, S., Xiao, H., Song, G., Zong, Z., Wang, L., Liu, Y., Li, H.: Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models. arXiv preprint arXiv:2403.16999 (2024)

Simon, H.A., Newell, A.: Human problem solving: The state of the theory in 1970. American Psychologist (1971)

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv:2302.13971 (2023)

Van Den Oord, A., Vinyals, O., et al.: Neural discrete representation learning (2017)

Wang, X., Huang, Q., Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.F., Wang, W.Y., Zhang, L.: Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In: CVPR. pp. 6629-6638 (2019)

Wang, X., Yu, Z., De Mello, S., Kautz, J., Anandkumar, A., Shen, C., Alvarez, J.M.: Freesolo: Learning to segment objects without annotations. In: CVPR. pp. 14176-14186 (2022)

Xia, Z., Han, D., Han, Y., Pan, X., Song, S., Huang, G.: Gsva: Generalized segmentation via multimodal large language models. arXiv:2312.10103 (2023)

Xu, J., De Mello, S., Liu, S., Byeon, W., Breuel, T., Kautz, J., Wang, X.: Groupvit: Semantic segmentation emerges from text supervision. In: CVPR. pp. 18134-18144 (2022)

Yang, S., Qu, T., Lai, X., Tian, Z., Peng, B., Liu, S., Jia, J.: An improved baseline for reasoning segmentation with large language model. arXiv:2312.17240 (2023)

Yang, S., Li, G., Yu, Y.: Dynamic graph attention for referring expression comprehension. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4644-4653 (2019)

Yang, Z., Ke, Z., Hancke, G., Lau, R.: Cross-domain semantic decoupling for weakly-supervised semantic segmentation (2023)

Yang, Z., Wang, J., Tang, Y., Chen, K., Zhao, H., Torr, P.H.: Lavt: Language-aware vision transformer for referring image segmentation. In: CVPR. pp. 18155-18165 (2022)

Ye, J., Chen, X., Xu, N., Zu, C., Shao, Z., Liu, S., Cui, Y ., Zhou, Z., Gong, C., Shen, Y ., et al.: A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv:2303.10420 (2023)

Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., Berg, T.L.: Mattnet: Modular attention network for referring expression comprehension. In: CVPR. pp. 1307-1315 (2018)

Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: ECCV. pp. 69-85 (2016)

Zhu, F., Zhu, Y., Chang, X., Liang, X.: Vision-language navigation with self-supervised auxiliary reasoning tasks. In: CVPR. pp. 10012-10022 (2020)

Zhu, L., Zhou, J., Liu, Y., Hao, X., Liu, W., Wang, X.: Weaksam: Segment anything meets weakly-supervised instance-level recognition. arXiv preprint arXiv:2402.14812 (2024)

Zong, Z., Ma, B., Shen, D., Song, G., Shao, H., Jiang, D., Li, H., Liu, Y.: Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404.13046 (2024)

A More Implementation Details

A.1 Generation of Target-related Cues

To obtain multiple target-related cues, we leverage the strong in-context capability of the Large Language Model (LLM) [16] to decompose the input referring expression and obtain the target-related textual cues. The Fig. 6 presents the LLM prompting details.

The prompt includes four parts: (1) general instruction P G , (2) output constraints P C , (3) incontext task examples P E , and (4) input question P Q . In part P G , we define a overall instruction for our task (i.e, decomposing the referring text) Then in part P C , we elaborate some details about the output (e.g, the sentence length for each cue description). In part P E , we curate

LLM

Referring text describes one object in a scene with

some states or attributes information. I hope you

can enumerate the state or attribute of the object.

Each output sentence include one state or attribute

and the first should given the referred object.

For example, given the referring text: woman in

white shirt looking down at laptop computer. You

should output: 1. a woman. 2. woman in white

shirt. 3. woman looking down at the computer.

Then given the text: {    }. Give your output:

a player wearing a blue and

gray uniform catches a ball

Text clues:

1. A player

2. He wears blue/gray uniform

3. He catches a ball

several in-context learning examples as guidance for the LLM to generating analogous output. Considering that the input referring expressions contain various sentence structures, in part P E the more examples given, the more reliable the output will be. The part P Q instructs the LLM to output the results given the input referring expression.

Q: a person in a wet suit stands in water

A: [a person].

[The person in a wet suit]

[The person standing in water]

Q: mug half full of beer sitting on the left of

table

A: [a mug].

[The mug half full of beer]

[The mug on the left of the table]

(a)

Q: an animal sticking its face through the

fence

A: [an animal]

[an animal sticking its face through the

fence]

Q: a woman in a black shit with her hands

under her chin

A: [a woman]

[The woman in a black shirt]

[The woman with her hands under her chin]

Q: aa woman in a red t - shirt with her

hand by her face

A: [a woman]

[The woman in red t - shirt]

[The woman with her hand by her face]

A: [a small napkin]

[The napkin with a cupcake printed on it]

[a small napkin]

(b)

(c)

(f)

(d)

(e)

Q: front left horse.

A: [a horse]

[The hose in front left]

Q: couch on left.

A: [a couch]

[The couch in left]

A: [a dog]

[the dog at top of picture]

(i)

(g)

(h)

Q: A small napkin with a cupcake printed on it

Q: dog at top of picture.

In Fig. 7, we give some examples of decomposing the referring expressions. The 'Q' denotes the input referring content and the 'A' denotes the answers of LLM. In most cases, we can obtain reliable target-related textual cues that do not contradict the original text input. Besides, we also notice that there are some cases in which the text is not sufficiently decomposed (e.g, the example (c)) or the LLM outputs redundant results (e.g, the example (f)), which hinders the model to benefit from progressive comprehension to some extent.

A.2 Text-to Image Classification Loss

Our work consists of multiple stages and utilizes L cls in TRIS [30] at each stage independently for response maps optimization. Here, we omit the index of stage n for clarity. L cls formulates the target localization problem as a classification process to differentiate between positive and negative text expressions. The key idea of L cls loss function is to contrast image-text pairs such that correlated image-text pairs have high similarity scores and uncorrelated image-text pairs have low similarity scores. While the referring text expressions for an image are used as positive expressions, the referring text expressions from other images can be used as negative expressions for this image. Thus, given a batch (i.e., B ) of image samples , each sample is mutually associated with one positive reference text (i.e., a text describing a specific object in the current image) and mutually exclusive with L negative reference texts (texts that are not related to the target object in the image). Note that the number of batches is equal to the sum of the positive samples and the negative samples (i.e., B = 1 + L ).

Speficially, in each training batch, B image-text pairs { I i , T i } B i =1 are sampled. Through the language and vision encoders, we can get referring embeddings Q ∈ R B × C and image embeddings V ∈ R B × H × W × C . Then, we obtain the response maps R ∈ R B × B × H × W by applying cosine similarity calculation and normalization operation. After the pooling operation as done in TRIS, we further obtain the alignment score matrix y ∈ R B × B . According to the L cls , for i th image in the batch, there is a prediction score y [ i, :] , where y [ i, i ] predicted by the corresponding text deserves a higher value (i.e, the positive one) and the others deserve lower values ( L negative ones). Then classification loss for the i th image from the batch can be formulated as cross-entropy loss:

The i denotes the index for the visual image and the j denotes the index for the referring text.

A.3 Referring Modulation Block …

In Sec. 3.2, we have mentioned that the conditional referring module (CRM) utilizes the decomposed textual cues to progressively modulate the referring embedding via a modulation block across N consecutive stages, and then produces the image-to-text response map by computing the patch-based similarity between visual and language embeddings.

Specifically, the modulation block is implemented by a vision-to-text and a textto-text cross-attention mechanism in cascade for facilitating the interaction between cross-modal features. In Fig. 8, we give an overview of the block design. For the block at each stage, We concatenate one targetrelated cue and the L negative text cues obtained from other images as the conditional text cues and then obtain the visionattended cue features by the vision-to-text attention. Then by learning the interaction between referring embedding and different

…

Linear layer

Figure 8: Illustration of referring modulation block. textual cue embeddings, the block is expected to enhance the integration of discriminative cues.

Attenti weight

A.4 Training and Inference

We implement our framework on PyTorch and train it for 15 epochs with a batch size of 36 ( i.e. , L +1 ) on a RTX4090 GPU with 24GB of memory. The input images are resized to 320 × 320. We use ResNet-50 [11] as our backbone of image encoder, and utilize the pre-trained CLIP [43] model to initialize the image and text encoders. The down-sampling ratio of visual feature s = 32 , the channels of vision feature C v = 2048 , text features C l = 1024 , and the unified hidden dimension C = 1024 . The network is optimized using the AdamW optimizer [37] with a weight decay of 1 e -2 and an initial learning rate of 5 e -5 with polynomial learning rate decay. For the LLM, we utilize the open-source powerful language model Mistral 7B [16] for referring text decomposition. For the proposal generator, we set the number of extracted proposals P = 40 for each image.

B More Quantitative Studies

B.1 Comparisons with other SOTA methods

In Tab. 6 and Tab. 7, we conduct the ablation studies about the measurement criterion of alignment score. The results demonstrate that the maximum value of the response map in each proposal better represents the alignment level of region-wise cross-modal alignment than the average

value. To validate the effectiveness of the modeling the progressive comprehension, we also quantitatively compare the outputs of our method at different stages in Tab. 8. The results show that the localization results gradually improve with more discriminative cues integration, especially in the early stages.

B.2 More Ablation Studies

Comparison between IaD loss and others. In our IaD loss L IaD , we adopt a hard assignment for deriving the loss function as GroupViT [54]. The motivation is that we aims to get the pseudo mask prediction by the accurate peak value point (i.e., the hard assignment results) instead of relying on whole score distribution S ( · ) (e.g., S a , S d in Sec. 3.4).

Thus utilizing the hard assignment to de- rive the IaD loss well matches our purpose, which helps rectify the ambiguous localization results. If we use the soft assignment (e.g., measuring KL divergence between S a and S d ), though the equivalent may be simpler, it not only does not match our purpose but also introduces more tricky com-

ponents for optimization (e.g., extra distribution regularization is required). In order to verify the argument, we conduct a comparison on RefCOCOg(G) val dataset as Tab. 9. The L IaD even causes a slight decline, while the proposed loss L KL brings clear improvement on localization accuracy.

Comparison between IaD loss and calibration loss in TRIS. There are essential differences between them. First, the calibration loss in TRIS [30] is used to suppress noisy background activation and thus help to re-calibrate the target response map. In contrast, in our method, we observe that, multiple referring texts corresponding to different instances in one image may locate the same instances (or we say overlapping), due to the lack of instance-level supervision in WRIS.

As for the implementation, the calibration loss adopts the global CLIP score of image-text to implement a simple contrastive learning for revising the response map. Differently, we simultaneously infer the response maps of different referring texts from the same image, and obtain the instance-level localization results by choos-

ing the mask proposal with max alignment score. To further verify the superiority of our loss, we conduct an ablation on the RefCOCO(val) dataset. We use the TRIS without calibration loss as the baseline and then separately introduced these two loss functions for comparison. Both ablations demonstrate that the IaD loss not only refines the response map (mIoU) but also significantly improves the localization accuracy (PointM).

C More Visualization of Localization Results

Progressive Comprehension for Localization. In Fig. 9, we also give visualization of each stage's response map for qualitative analysis. The results show that our proposed CRM module can effectively integrate the target-related textual cues. For example, in the first row, the method produces ambiguous localization result at the first stage. After taking the cue 'with gold necklace' into consideration, the attention is transferred to the target object at the second stage. Finally, after considering all the cues, the method produces less ambiguous and more accurate localization results.

Q: woman with gold necklace sitting behind little birthday girl

Q1:

a base ball player

Q2:

He kneels down…

Q3:

NULL.

Q1:

a woman

Q2:

She is with…

Q3:

She sits behind …

Q1:

a young man

Q2:

He is with a shirt

Q3:

The shirt has a …

Q1:

a boy

Q2:

a boy in black ..

Q3:

The boy is bending ..

COCO_train2014_0

COCO_train2014_000000095018

a child with a black shirt and blue

Qualitative Comparison with Other method. More qualitative comparisons of our method with other methods are shown in the Fig. 10. For the example shown in the fifth row, the query is 'a man in a arm striped sweater'. The TRIS [30] mistakenly locates the left man as the target regions. In contrast, our PCNet optimizes the response map generation process by continuously modulating the referring embedding query conditioned on the target-related cues instead of a static referring embedding. As a result, our method can obtain more accurate localization result.

D Proposal Generator

In this work, we adopt the two representative pre-trained segmentors: FreeSOLO [52] and SAM [20] for proposal generation. Specifically, the FreeSOLO [52] is a fully unsupervised learning method that learns class-agnostic instance segmentation without any annotations. The SAM's training utilizes densely labeled data, but it does not include semantic supervision. This supervision does not contradict our weakly supervised RIS setting. More importantly, it offers a promising solution as an image segmentation foundation model and can be used for refining the coarse localization results from weakly-supervised methods into precise segmentation masks as done in the recent works [10, 63]. For the usage of SAM, We adopt the ViT-H backbone, the hyperparameter predicted iou threshold and stability score threshold are set to 0.7, and points per side is set to 8.

In Fig. 11 and Fig. 12, we give the generated mask proposals examples by FreeSOLO [52] and SAM [20], respectively. We notice that there are often overlaps among the generated proposals. Thus, we refine the generated proposals by filtering out candidate proposals with small area (the threshold is set as 1000 ) and then selecting the ones with smaller intersection over union (the threshold is set as 0 . 8 ). Considering that the number of proposals generated by the segmentor may be different for different image inputs, in implementation, we maintain consistency by selecting the top 40 proposals with the largest area ( P = 40 ). If fewer than 40 , we simply complete it with an all-zero mask.

E Broader Impacts

While we do not foresee our method causing any direct negative societal impact, it may potentially be leveraged by malicious parties to create applications that could misuse the segmentation capabilities for unethical or illegal purposes. We urge the readers to limit the usage of this work to legal use cases.

Q: '

a catcher rushing to make a play on the ball'

Q: '

a boy in black t - shirt and jeans bending by keeping his hands on knees'

Q : ' a smiling man with a small infant wearing a beige t - shirt and gray jeans'

Q: '

a man in a arm striped sweater'

Q: '

a postcard with picture of face of cute girl'

Q: '

dark haired woman wearing a blue jacket next to a teddy bear'

(a) Image

(e) GT

(b) TRIS

(d) PCNet

Roud

A

一

T

(e) First Stage

色

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In Sec. 1, we make the claims to reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Sec. 5, we discuss the limitations of the work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide implementation details in Sec. 4 and the Appendix for reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will consider releasing the data and code once the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide details about the experimental setting (e.g., hyperparameters) in Sec. 4 and the Appendix for reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We do not report error bars due to limited computing resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information on the computer resources in the implementation details of Sec. 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses societal impacts of the work in the Appendix Sec. E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper properly cites the existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAiming for the WRIS task, the work proposes a novel framework to leverage target-related textual cues from the description for progressively localizing the target object. The authors use a Large Language Model (LLM) to decompose the input text description into short phrases which are taken as target-related cues and fed into a Conditional Referring module. Besides, Region-aware Shrinking (RaS) loss and Instance-aware Disambiguation (IaD) loss are proposed to facilitate fine-grained cross-modal alignment.\n\nSTRENGTHS:\n1. The motivation is clear and easy to understand. The idea of leveraging target-related textual cues decomposed by LLM for progressive localization makes sense.\n\n2. This work develops multiple consecutive Conditional Referring Modules. RaS and IaD two loss objectives are seamlessly integrated into the framework.\n\n3.  It is novel to implement RaS and IaD two loss objectives by harnessing the capabilities of segmentation foundation models. The SAM[1] does not include strong semantic prior knowledge like GroundingDINO and Grounded-SAM[2], and it is suitable for the weakly-supervised setting.\n\n4. The work shows outstanding object localization ability and outperforms counterparts on three common benchmarks. Especially after refined by the SAM, the results are promising.\n\n[1] Segment anything ICCV2023\n\n[2] Grounding dino: Marrying dino with grounded pre-training for open-set object detection\n\nWEAKNESSES:\n1. In table 1, only the results refined by SAM[1] are reported. Considering that FreeSOLO [2] is an unsupervised segmentation model, I am curious about the results refined by FreeSOLO. \n\n2.  This work introduces multiple stages for progressive localization. It may increase the training and inference time of the model.  The time comparison of model training and inference may be needed.\n\n[1] Segment anything ICCV2023\n\n[2] Freesolo: Learning to segment objects without annotations CVPR2022\n\nQUESTIONS:\n1. The mask proposals by SAM are often part of the whole instance.  It will affect the refined results. Does this work do anything special to alleviate the problem? The author can give more details about this issue.\n\n2. Table 1 shows the results of the pseudo labels obtained at the wealy-supervised training stage. Can the authors give the results after pseudo labels based on supervised training?\n\nLIMITATIONS:\nNaN", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-02"}

comments: SUMMARY:
This paper proposes the Progressive Comprehension Network (PCNet) for the WRIS task. this model achieves visual localization by progressively incorporating target-related textual cues for visual-linguistic alignment. Although experimental results have demonstrated the effectiveness of this paper, several aspects of the paper lack clarity.

STRENGTHS:
1.	The corresponding experiments validate the effectiveness of the proposed method on three popular benchmarks. the proposed method outperforms existing methods.
2.	The two loss functions introduced in this paper effectively enhance the accuracy of the response maps.

WEAKNESSES:
1.	The Conditional Referring Module (CRM) is implemented through multiple cross-attentions, which is a relatively common approach, lacking novelty. 
2.	What is the general idea of classification loss mentioned in the paper? Please explain the fundamental research insights of this work.
3.	In the Region-aware Shrinking (RaS) loss, what does the ⊙ symbol represent in Equation 5? Please clarify its meaning. 
4.	This article only conducted ablation experiments on the RefCOCOg dataset, so the ablation experiments are insufficient. It is recommended to validate on multiple datasets (such as RefCOCO+, RefCOCO) to comprehensively evaluate the effectiveness of our method.
5.	The order of the tables is somewhat confusing, with Table 9 preceding Table 6. It is recommended to reorder the tables to make them more in line with logical sequence or reading habits.
6.	Which datasets were the ablation experiments in Tables 7-9 conducted on? Please specify.

QUESTIONS:
None

LIMITATIONS:
See the weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInspired by human's step-by-step cognitive process for localizing a target object in an image, the paper proposes Progressive Comprehension Network (PCNet) for the task of weakly-supervised referring image segmentation (WRIS) where text is the only supervision signal. PCNet first decomposes a long, complex text description into multiple target-related cues using a large language model (LLM), and the specialized designed module, Conditional Referring Module (CRM), progressively refines the text-to-image response map using the generated text cues in multiple stages. Region-aware Shrinking (RaS) is introduced for constraining the latter stage to have a more compact, target-related response map, and Instance-aware Disambiguation (IaD) loss is proposed for differentiating between the target object and the other objects in the input image. The experiment results show the superiority of the proposed method.\n\nSTRENGTHS:\n- The paper provides extensive experiment results proving the validity of the proposed method, PCNet, and also ablation studies showing the contributions of main components of the model.\n- Figures help readers understand the proposed method.\n\nWEAKNESSES:\n- The proposed method uses the contrastive loss of TRIS, but not generates independent response maps from a positive cue and L negative cues. In Equation (3) and (4), it integrates all the text cue features and generates a single response map R_n that contains information from both positive and negative cues. This does not make sense. How did the authors differentiate between positive and negative cues when computing the classification loss L_Cls? (This is why I rated \"poor\" for presentation).\n- IaD, the loss to differentiate between the target object and other objects in the same image is similar to the calibration loss used in TRIS; the idea is the same while the calibration loss uses CLIP image-text similarity for differentiation. The authors should have discussed about this loss in the paper, and also done an ablation study comparing the two losses.\n- The PCNet uses the mask proposals generated by a pre-defined segmentation mask generator such as FreeSOLO and SAM. We cannot ignore the possibility of knowledge being distilled from the mask generator into PCNet, especially through RaS and IaD losses, main contributions of the paper. More specifically, we cannot be sure that the performance gains from using RaS and IaD do not come from the mask generator's knowledge.\n\nQUESTIONS:\nPlease rebut the above weaknesses. I also have minor questions:\n- In Figure 2, the arrow comes from Q_{n+1} to Cls. How is Q_{n+t} used for computing the classification loss?\n- How did the authors sample L negative textual cues? one cue from each of the examples in the same mini-batch?\n- I am curious about the rationale behind Equations (2) and (3). Why did the authors use residual connection only after the mlp layer, not also after the self-attention layer like in Transformer?\n- Does the order in which text cues are used affect the performance of the model? E.g., is there a performance difference between feeding into the CRM module in the order of q_0, q_1, q_2 and in the order of q_0, q_2, q_1?\n\nLIMITATIONS:\nThe paper addressed the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Progressive Comprehension Network for weakly-supervised referring image segmentation, which mimics the human process of progressive understanding by breaking down sentences into segments and gradually narrowing down the target range. The main contributions include: A multi-stage Conditional Referring Module to progressively comprehend text cues. A Region-aware Shrinking Loss to constrain the target region to gradually shrink. An Instance-aware Disambiguation Loss to eliminate overlap between different instances.\n\nSTRENGTHS:\n1. Presentation: The majority of the paper is well-written and easy to follow, providing clear explanations of the key concepts.\n2. Motivation: The motivation is reasonable and convincing. By mimicking the human process of understanding concepts from coarse to fine, the method progressively refines the final mask. Directly requiring the mask to continuously shrink could lead to trivial solutions, and two losses are introduced to further support this pipeline. The proposed method aligns well with the motivation.\n3. Experiments: Comprehensive ablation experiments demonstrate the effectiveness of the proposed modules\n\nWEAKNESSES:\n1. The concept of mimicking the human process of understanding concepts from coarse to fine-grained, along with multi-stage refinement for different parts of sentences, has been explored in prior REC research [1]. This approach involves parsing sentences into multiple parts and conducting multi-stage refinement. Applying REC's ideas [1] to WRIS can also be considered a contribution, but it should be thoroughly discussed in comparison to [1], highlighting any differences, and cited appropriately.\n2. There is a lack of discussion and citation of recent WRIS work, such as [2], in related work and the main text. It is necessary to discuss and cite the latest WRIS work in both the related work section and the main table.\n\n3. Section 3.4 is not entirely understandable. I suggest the authors re-describe this section to make it clearer and easier to understand.\n  \n4. The top of Fig.2 is not clear enough, especially the subscripts a and d, I suggest the author re-design this part.\n\n\n [1] Yang S, Li G, Yu Y. Dynamic graph attention for referring expression comprehension[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 4644-4653.\n\n [2] Dai Q, Yang S. Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13711-13722.\n\nQUESTIONS:\n1. Why is the range of Equation 6 set to [0,1]? Can the combination of other proposals (denoted as 𝑚_𝑏 ) be simply approximated by the complement of the foreground, since the proposals segmented by SAM nearly cover the entire image? In this case, if IoU(𝑅_𝑛,𝑚_𝑓) < IoU(𝑅_𝑛,𝑚_𝑏), wouldn't Equation 6 be greater than 1?\n2. Does 𝑚_𝑓 change at different stages in Equation 7, for instance, when different proposals are selected at different stages? What issues could this cause? In Equation 7, are 𝑛 and 𝑛+1 reversed? According to Section 3.3, the ambiguity score should be lower at later stages.\n3. Section 3.1 mentions that there are five phrases for each sentence, and Section 3.2 states that each stage uses one text cue, so there should be five stages. Why do the implementation details mention only three stages?\n4. In Section 3.4, how do you sample extra 𝑁_𝑑 texts? Are they randomly sampled from multiple text descriptions corresponding to the same image? How do you ensure that the sampled text descriptions correspond to different target referents? Because in refcoco series dataset, one referent may have multiple text expressions.\n5. In the bottom part of Tab.1, are the numbers all obtained by using a single peak point as the prompt for SAM? Did you try using the response map as the prompt for SAM?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}