Boosting Weakly Supervised Referring Image Segmentation via Progressive Comprehension

Authors: zaiquan  yang
yuhao  liu
jiaying  lin
gerhard  hancke
rynson w h lau
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-27 05:51:30

Abstract: This paper explores the weakly-supervised referring image segmentation (WRIS) problem, and focuses on a challenging setup where target localization is learned directly from image-text pairs. 
We note that the input text description typically already contains detailed information on how to localize the target object, and we also observe that humans often follow a step-by-step comprehension process (\ie, progressively utilizing target-related attributes and relations as cues) to identify the target object. 
Hence, we propose a novel Progressive Comprehension Network (PCNet) to leverage target-related textual cues from the input description for progressively localizing the target object.
Specifically, we first use a Large Language Model (LLM) to decompose the input text description into short phrases. These short phrases are taken as target-related cues and fed into a Conditional Referring Module (CRM) in multiple stages, to allow updating the referring text embedding and enhance the response map for target localization in a multi-stage manner.
Based on the CRM, we then propose a Region-aware Shrinking (RaS) loss to constrain the visual localization to be conducted progressively in a coarse-to-fine manner across different stages.
Finally, we introduce an Instance-aware Disambiguation (IaD) loss to suppress instance localization ambiguity by differentiating overlapping response maps generated by different referring texts on the same image. 
Extensive experiments show that our method outperforms SOTA methods on three common benchmarks.

Main paper: 
 introduction: 
 Referring Image Segmentation (RIS) aims to segment a target object in an image via a user-specified input text description. RIS has various applications, such as text-based image editing [17, 13, 2] and human-computer interaction [62, 51] . Despite remarkable progress, most existing RIS works [7, 58, 27, 26, 21, 5] rely heavily on pixel-level ground-truth masks to learn visual-linguistic alignment. Recently, there has been a surge in interest in developing weakly-supervised RIS (WRIS) methods via weak supervisions, e.g., bounding-boxes [9] , and text descriptions [54, 18, 30, 4] , to alleviate burden of data annotations. In this work, we focus on obtaining supervision from text descriptions only.The relatively weak constraint of utilizing text alone as supervision makes visual-linguistic alignment particularly challenging. There are some attempts [30, 18, 22, 46] to explore various alignment workflows. For example, TRIS [30] classifies referring texts that describe the target object as positive texts while other texts as negative ones, to model a text-to-image response map for locating potential target objects. SAG [18] introduces a bottom-up and top-down attention framework to discover Q2: "blue and gray uniform" + Figure 1 : Given an image and a language description as inputs (a), RIS aims to predict the target object (d). Unlike existing methods (e.g., TRIS [30] (e) -a WRIS method) that directly utilize the complete language description for target localization, we observe that humans would naturally break down the sentence into several key cues (e.g., Q1 -Q3) and progressively converge onto the target object (from (b) to (d). This behavior inspires us to develop the Progressive Comprehension Network (PCNet), which merges text cues pertinent to the target object step-by-step (from (f) to (h)), significantly enhancing visual localization. ⊕ denotes the text combination operation.individual entities and then combine these entities as the target of the referring expression. However, these methods encode the entire referring text as a single language embedding. They can easily overlook some critical cues related to the target object in the text description, leading to localization ambiguity and even errors. For example, in Fig. 1 (e), TRIS [30] erroneously activates all three players due to its use of cross-modality interactions between the image and the complete language embedding only.We observe that humans typically localize target objects through a step-by-step comprehension process. Cognitive neuroscience studies [48, 41] also support this observation, indicating that humans tend to simplify a complex problem by breaking it down into manageable sub-problems and reasoning them progressively. For example, in Fig. 1(b-d ), human perception would first begin with "a player" and identify all three players (b). The focus is then refined by the additional detail "blue and gray uniform", which helps exclude the white player on the left (c). Finally, the action "catches a ball" helps further exclude the person on the right, leaving the correct target person in the middle (d).Inspired by the human comprehension process, we propose in this paper a novel Progressive Comprehension Network (PCNet) for WRIS. We first employ a Large Language Model (LLM) [59] to dissect the input text description into multiple short phrases. These decomposed phrases are considered as target-related cues and fed into a novel Conditional Referring Module (CRM), which helps update the global referring embedding and enhance target localization in a multi-stage manner. We also propose a novel Region-aware Shrinking (RaS) loss to facilitate visual localization across different stages at the region level. ReS first separates the target-related response map (indicating the foreground region) from the target-irrelevant response map (indicating the background region), and then constrains the background response map to progressively attenuate, thus enhancing the localization accuracy of the foreground region. Finally, we notice that salient objects in an image can sometimes trigger incorrect response map activation for text descriptions that aim for other target objects. Hence, we introduce an Instance-aware Disambiguation (IaD) loss to reduce the overlapping of the response maps by rectifying the alignment score of different referring texts to the same object.In summary, our main contributions are as follows :• We propose the Progressive Comprehension Network (PCNet) for the WRIS task. Inspired by the human comprehension processes, this model achieves visual localization by progressively incorporating target-related textual cues for visual-linguistic alignment.• Our method has three main technical novelties. First, we propose a Conditional Referring Module (CRM) to model the response maps through multiple stages for localization. Second, we propose a Region-aware Shrinking (RaS) loss to constrain the response maps across different stages for better cross-modal alignment. Third, to rectify overlapping localizations, we propose an Instance-aware Disambiguation (IaD) loss for different referring texts paired with the same image.• We conduct extensive experiments on three popular benchmarks, demonstrating that our method outperforms existing methods by a remarkable margin. 
related work: 
 Referring Image Segmentation (RIS) aims to segment the target object from the input image according to the input natural language expression. Hu et al. [14] proposes the first CNN-based RIS method. There are many follow-up works. Early methods [60, 28, 38] focus on object-level cross-modal alignment between the visual region and the corresponding referring expression. Later, many works explore the use of attention mechanisms [15, 7, 58, 19] or transformer architectures [58, 29] to model long-range dependencies, which can facilitate pixel-level cross-model alignment.For example, CMPC [15] employs a two-stage progressive comprehension model to first perceive all relevant instances through entity wording and then use relational wording to highlight the referent. In contrast, our approach leverages LLMs to decompose text descriptions into short phrases related to the target object, focusing on sentence-level (rather than word-level) comprehension, which aligns more closely with human cognition. Focusing on the visual grounding, DGA [56] also adopts multi-stage refinement. It aims to model visual reasoning on top of the relationships among the objects in the image. Differently, our work addresses the weakly RIS task and aims to alleviate the localization ambiguity by progressively integrating fine-grained attribute cues.Weakly-supervised RIS (WRIS) has recently begun to attract some attention, as it can substantially reduce the burden of data labeling especially on the segmentation field [25, 57, 63] . Feng et al. [9] proposes the first WRIS method, which uses bounding boxes for annotations. Several subsequent works [18, 22, 30] attempt to use weaker supervision signal, i.e., text descriptions. SAG [18] proposes to first divide image features into individual entities via bottom-up attention and then employ top-down attention to learn relations for combining entities. Lee et al. [22] generate Grad-CAM for each word of the description and then consider the relations using an intra-chunk and inter-chunk consistency. Instead of merging individual responses, TRIS [30] directly learns the text-to-image response map by contrasting target-related positive and target-unrelated negative texts. Inspired by the generalization capabilities of segmentation foundation models [20, 8, 34] , PPT [6] enables effective integration with pre-trained language-image models [43, 33] and SAM [20] by a lightweight point generator to identify the referent and context noise. Despite their success, these methods encode the full text as a single embedding for cross-modality alignment, which overlooks target-related nuances in the textual descriptions. In contrast, our method proposes to combine progressive text comprehension and object-centric visual localization to obtain better fine-grained cross-modal alignment.Large Language Models (LLMs) are revolutionizing various visual domains, benefited by their userfriendly interfaces and strong zero-shot prompting capabilities [3, 49, 1, 47] . Building on this trend, recent works [42, 55, 53, 45, 64] explore the integration of LLMs into vision tasks (e.g., languageguided segmentation [55, 53] , relation [23] , and image classification [42] ) through parameter-efficient fine-tuning or knowledge extraction. For example, LISA [55] and GSVA [53] utilize LLaVA [32] , a large vision-language model (LVLM), as a feature encoder to extract visual-linguistic crossmodality features and introduce a small set of trainable parameters to prompt SAM [20] for reasoning segmentation. RECODE [23] and CuPL [42] leverage the knowledge in LLMs to generate informative descriptions as prompts for different categories classification. Unlike these works, we capitalize on the prompt capability of LLMs to help decompose a single referring description into multiple target object-related phrases, which are then used in our progressive comprehension process for RIS.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: In Sec. 5, we discuss the limitations of the work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
our method: 
 In this work, we observe that when identifying an object based on a description, humans tend to first pinpoint multiple relevant objects and then narrow their focus to the target through step-by-step reasoning [48, 41] . Inspired by this, we propose a Progressive Comprehension Network (PCNet) for WRIS, which enhances cross-modality alignment by progressively integrating target-related text cues at multiple stages. Fig. 2 shows the overall framework of our PCNet.Given an image I and a referring expression T as input, we first feed T into a Large Language Model (LLM) to break it down into K short phrases T sub = {t 0 , t 1 , • • • , t K-1 }, referred to as target-related text cues. We then feed image I and referring expression T and the set of short phrases T sub into image encoder and text encoder to obtain visual feature V 0 ∈ R H×W ×Cv and languageStage 0 Q sub = {q 0 , q 1 , • • • , q K-1 }, with q k ∈ R 1×Ct, where H = H I /s and W = W I /s. C v and C l denote the numbers of channels of visual and text features. s is the ratio of down-sampling. We then use projector layers to transform the visual feature V 0 and textual features Q 0 and Q sub to a unified dimension C, i.e., V 0 ∈ R H×W ×C , Q 0 and q i are in R 1×C .We design PCNet with multiple consecutive Conditional Referring Modules (CRMs) to progressively locate the target object across N stages 1 . Specifically, at stage n, the n-th CRM updates the referring embedding Q n into Q n+1 conditioned on the short phrase q n from the proposed Referring Modulation block. Both Q n+1 and visual embedding V n are fed into Response Map Generation to generate the text-to-image response map R n and updated visual embedding V n+1 . Finally, the response map R N -1 generated by the n-th CRM is used as the final localization result. To optimize the resulting response map for accurate visual localization, we employ the pre-trained proposal generator to obtain localization-matched mask proposals. We also propose Region-aware Shrinking (RaS) loss to constrain the visual localization in a coarse-to-fine manner, and Instance-aware Disambiguation (IaD) loss to suppress instance localization ambiguity.In the following subsections, we first discuss how we decompose the input referring expression into target-related cues in Sec. 3.1. We then introduce the CRM in Sec. 3.2. Finally, we present our Region-aware Shrinking loss in Sec. 3.3, and Instance-aware Disambiguation loss in Sec. 3.4.Existing works typically encode the entire input referring text description, and can easily overlook some critical cues (e.g., attributes and relations) in the description (particularly for a long/complex description), leading to target localization problems. To address this problem, we propose dividing the input description into short phrases to process it individually. To do this, we leverage the strong in-context capability of the LLM [1] to decompose the text description. We design a prompt, with four parts, to instruct the LLM to do this: (1) general instruction P G ; (2) output constraints P C ;(3) in-context task examples P E ; and (4) input question P Q . P G describes the overall instruction, e.g."decomposing the referring text into target object-related short phrases". P C elaborates the output setting, e.g., sentence length of each short phrase. In P E , we specifically curate several in-context pairs as guidance for the LLM to generate analogous outputs. Finally, P Q encapsulates the input text description and the instruction words for the LLM to execute the operation. The process of generating target-related cues is formulated as:EQUATIONwhere K represents the total number of phrases, which varies depending on the input description. Typically, longer descriptions more likely yield more phrases. To maintain consistency in our training dataset, we standardize it to five phrases (i.e., K = 5). If fewer than five phrases are produced, we simply duplicate some of the short phrases to obtain five short phrases. In this way, phrases generated by LLM are related to the target object and align closely with our objective.Given the decomposed phrases (i.e., target-related cues), we propose a CRM to enhance the discriminative ability on the target object region conditioned on these phrases, thereby improving localization accuracy. As shown in Fig. 2 , the CRM operates across N consecutive stages. At each stage, it first utilizes a different target-related cue to modulate the global referring embedding via a referring modulation block and then produces the image-to-text response map through a response map generation block.Referring Modulation Block. Considering the situation at stage n, we first concatenate one targetrelated cue q n and the L negative text cues obtained from other images2 , to form q ′ n ∈ R (L+1)×C . We then fuse the visual features V n with q ′ n through a vision-to-text cross-attention, to obtain vision-attended cue features qn ∈ R (L+1)×C , as:EQUATIONwhere A v→t ∈ R (L+1)×H×W denotes the vision-to-text inter-modality attention weight. W V * and W q ′ * are learnable projection layers. ⊗ denotes matrix multiplication. Using the vision-attended cue features qn , we then enrich the global textual features Q n into cue-enhanced textual features Q n+1 ∈ R 1×C through another text-to-text cross-attention, as:EQUATIONwhere ) represents the text-to-text intra-modality attention weight. W Q * and W q * are learnable projection layers. In this way, we can enhance the attention of Q n on the target object by conditioning its own target-related cue features and the global visual features.A t→t ∈ R 1×(L+1Response Map Generation. To compute the response map, we first update the visual features V n to Vn by integrating them with the updated referring text embedding Q n+1 using a text-to-visual cross-attention, thereby reducing the cross-modality discrepancy. Note that Vn is then used in the next stage (i.e., V n+1 = Vn ). The response map R n ∈ R H×W at the n-th stage is computed as:EQUATIONwhere Norm normalizes the output in the range of [0,1]. To achieve global visual-linguistic alignment, we adopt classification loss L Cls in [30] to optimize the generation of the response map at each stage. It formulates the target localization problem as a classification process to differentiate between positive and negative text expressions. While the referring text expressions for an image are used as positive expressions, the ones from other images can be used as negative for this image. More explanations are given in appendix.Despite modulating the referring attention with the target-related cues stage-by-stage, image-text classification often activates irrelevant background objects due to its reliance on global and coarse response map constraints. Ideally, as the number of target-related cues used increases across each stage, the response map should become more compact and accurate. However, directly constraining the latter stage to have a more compact spatial activation than the former stage can lead to a trivial solution (i.e., without target activation). To address this problem, we propose a novel region-aware shrinking (RaS) loss, which segments the response map into foreground (target) and background (non-target) regions. Through contrastive enhancement between these regions, our method gradually reduces the background interference while refining the foreground activation in the response map.Specifically, at stage n, we first employ a pretrained proposal generator to obtain a set of mask proposals,M = {m 1 , m 2 , • • • , m P },where each proposal m p is in R H×W and P is the total number of segment proposals. We then compute a alignment score between the response map R n and each proposal m p in M as:EQUATIONwhere ⊙ denotes the hadamard product. The proposal with the highest score (denoted as m f ) is then treated as the target foreground region, while the combination of other proposals (denoted as m b ) is regarded as non-target background regions. With the separated regions, we define a localization ambiguity S amb n , which measures the uncertainty of the target object localization in the current stage n, as:EQUATIONwhere S amb n is in the range of [0, 1], and IoU denotes the intersection over union. When the localization result (i.e., the response map) matches the only target object proposal instance exactly, ambiguity is 0. Conversely, if it matches the more background proposals, ambiguity approaches 1.Assuming that each target in the image corresponds to an instance, by integrating more cues, the model will produce a more compact response map and gradually reduce the ambiguity. Consequently, based on the visual localization results from two consecutive stages, we can formulate the region-aware shrinking objective for a total of N stages as:EQUATIONBy introducing region-wise ambiguity, L RaS can direct non-target regions to converge towards attenuation while maintaining and improving the quality of the response map in the target region. This enables the efficient integration of target-related textual cues for progressively finer cross-modal alignment. Additionally, the mask proposals can also provide a shape prior to the target region, which helps to further enhance the accuracy of the target object localization.Although the RaS loss can help improve the localization accuracy by reducing region-wise ambiguity within one single response map, it takes less consideration of the relation between different instancewise response maps. Particularly, we note that, given different referring descriptions that refer to different objects of an image, there are usually some overlaps among the corresponding response maps. For example, in Fig. 2 T d = {t 1 , t 2 , • • • , t N d },where the referred target object of each text description t d is in the image I a but is different from the target object referred to by T a . We then obtain the image-to-text response maps R a andR d = {R 1 , R 2 , • • • , R N d }for T a and T d through Eq. ( 4).Here, we omit the stage index n for clarity. Then, based on the Eq. ( 5), we obtain the alignment scores S a and {S d } N d d=1 for T a and T d . In S, the larger the value, the higher the alignment between the corresponding proposal (specified by the index) and the current text. To disambiguate overlapping activated regions, we constrain that the maximum index of the alignment score between S a and each of S d must be different from each other (i.e., different texts must activate different objects). Here, we follow [50] to compute the index vector, y ∈ R 1×P , as: EQUATIONEQUATION)By enforcing the constraint at each stage, the response maps activated by different referring descriptions in an image for different instances are separated, and the comprehension of the discriminative cues is further enhanced.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
settings: 
 Dataset. We have conducted experiments on three standard benchmarks: RefCOCO [61] , RefCOCO+ [61] , and RefCOCOg [39] . They are constructed based on MSCOCO [24] . Specially, the referring expressions in RefCOCO and RefCOCO+ focus more on object positions and appearances, respectively, and they are characterized by succinct descriptions, averaging 3.5 words in length. RefCOCOg contains much longer sentences (average length of 8.4 words), making it more challenging than others. RefCOCOg includes two partitions: UMD [40] and Google [39] .Implementation Details. We train our framwork for 15 epochs with a batch size of 36 on an RTX4090 GPU. The total loss for training is L total = L Cls + L RaS + L IaD . By default, we set the number of stages N to 3, and the number of the additional text descriptions sampled for each image N d to 1. Without loss of generality, we use FreeSOLO [52] and SAM [20] as the proposal generators to obtain two versions: PCNet S and PCNet F . Refer to Sec. A for more implementation details."a child sitting on a chair with a blue shirt on" "a woman wearing a green shirt and black pants""the zebra that is behind the front zebra and the elephant""zebra looking at the camera through the legs of an elephant" "base ball player ready to swing""man squatting in a head gear between the batter and umpire" "a nab wearing a white shirt and tie" "a white -haired man talking on a phone"Referring Expressions:Referring Expressions: Referring Expressions:Referring Expressions: Evaluation Metrics. We argue that the key to WRIS is target localization, and the performance evaluation should not rely primarily on pixel-wise metrics. With the accurate localization points, pixel-level masks can be readily obtained by prompting the pre-trained segmentors (e.g., SAM [20] ). Thus, following [18, 22, 30] , we adopt localization-based metric (i.e., PointM), and pixel-wise metrics (i.e., mean and overall intersection-over-union (mIoU and oIoU) for evaluation. PointM [30] is used to evaluate the localization accuracy, which computes the ratio of activation peaks in the mask region.Q: "3Quantitative comparison. Tab. 1 compares our method with various SOTA methods. Specifically, we first compare target localization accuracy using the PointM metric. We evaluate two model variants: PCNet F and PCNet S , which use different segmentors (e.g., FreeSOLO [52] and SAM [20] ) to extract mask proposals for RaS and IaD losses. Even when using FreeSOLO as the proposal generator, our model still significantly outperforms all compared methods. For example, on the most challenging dataset, RefCOCOg, with more complex object relationships and longer texts, PCNet F achieves performance improvements of 55.2% and 10.3% on the Val (G) set compared to SAG and TRIS 3 . PCNet S further boosts the performance if we replace FreeSOLO with the stronger SAM.In addition, we verify the accuracy of the response map through pixel-wise mIoU metric. Results are shown in the middle part of Tab. 1. Our PCNet still achieves superior performances on all benchmarks, against all compared methods. Particularly, PCNet F and PCNet S outperform TRIS by an improvement of 10.8% and 14.1% mIoU, respectively, on the RefCOCOg Val (G) set. In the bottom part of Tab. 1, we compare the accuracy of the extracted mask proposals generated using the target localization point (i.e., the peak point of the response map) to prompt SAM. We can see that our PCNet significantly outperforms other WRIS methods. We can also see that higher PointM values correlate with higher mIoU accuracy values of the corresponding mask proposals for different methods. We further tested the mask accuracy using the Ground-Truth localization point (i.e., the last row), and find that its performance even surpasses the fully-supervised method, LAVT [58] . All these results highlight the critical importance of target localization (i.e., peak point) for the WRIS task.In Fig. 3 , we show some visual results of our method across different scenes by using the target localization point (i.e., the green marker) to prompt SAM to generate the target mask. Our method effectively localizes the target instance among other instances within the image, even in complex scenarios with region occlusion (a), multiple instances (b), similar appearance (c), and dim light (d).We conduct ablation experiments on the RefCOCOg dataset and report the results on the Val (G) set from both PCNet S and PCNet F in Tab. 2, and from PCNet F in Tab. 3, Tab. 4, and Tab. 5.Component Analysis. In Tab. 2, we first construct a single-stage baseline (1st row) to optimize visual-linguistic alignment by removing all proposed components and using only the global imagetext classification loss L Cls . We then introduce the proposed conditional referring module (CRM) to the baseline to allow for multi-stage progressive comprehension (2nd row). To validate the efficacy of the region-aware shrinking loss (RaS) and instance-aware disambiguation loss (IaD), we introduce them separately (3rd and 4th rows). Finally, we combine all proposed components (5th row).h The results demonstrate that ❶ even using only L Cls , progressively introducing target-related cues through CRM can still significantly enhance target object localization. In particular, PCNet S achieves improvements of 3.1% on PointM and 5.9% on mIoU; ❷ by using L RaS to constrain response maps, making them increasingly compact and complete during the progressive comprehension process, the accuracy of target localization is dramatically enhanced, resulting in an improvement of 11.6% on PointM. ❸ although L IaD can facilitate the separation of overlapping response maps between different instances within the same image and improve the discriminative ability of our model on the target object, the lack of constraints between consecutive stages results in a smaller performance improvement than L RaS ; and ❹ all components are essential for our final PCNet, and combining them achieves the best performance. In Fig. 4 , we also provide the visual results of the ablation study on two examples. We can see that each component can bring obvious localization improvement.Number of Iterative Stages. In Tab. 3, we analyze the effect of the number of iterative stages N .When N = 1, we can only apply L Cls and L IaD , but not L RaS , resulting in inferior results. Increasing N from 1 to 2 significantly improves the performance due to the progressive introduction of targetrelated cues. However, the improvement from N = 2 to N = 3 is less pronounced than from N = 1 to N = 2, and the performance stabilizes at N = 3. At N = 4, the performance slightly declines. This is because when the effective short-phrases decomposed by LLM are fewer than the number of stages, we need to repeat text phrases in later stages, which may affect the loss optimization.Modulation Strategy. In Tab. 4, we ablate different variants of CRM: ❶ directly adding target cue features q n and global referring features Q n (denoted as ADD); ❷ fusing q n and Q n using only text-to-text cross-attention (denoted as TTA); ❸ first employing a vision-to-text cross-attention to fuse visual features V n and q n to obtain vision-attended features qn , and then adding them to Q n (denoted VTA+ADD). The results demonstrate that ADD is the least efficient method. TTA Q: a young man with a shirt that has a giant musical note on it Q: a boy in black t -shirt and jeans bending by keeping his hands on knees Q: a white mixture with herbs that is spread on top of some doughQ: "a catcher rushing to make a play on the ball" Our method does have limitations. For example, as shown in Fig. 5 , when the text description refers to multiple objects, our method fails to return all referring regions. This is because our model design always assumes that there is only one object referred to by the language expression. In the future, we plan to incorporate more fine-grained vision priors [35, 44] and open-world referring descriptions (e.g., camouflaged [12] , semi-transparent [31] , shadow [36] and etc.) into the model design to enable a more generalized solution.A More Implementation DetailsTo obtain multiple target-related cues, we leverage the strong in-context capability of the Large Language Model (LLM) [16] to decompose the input referring expression and obtain the target-related textual cues. The Fig. 6 presents the LLM prompting details. In Fig. 7 , we give some examples of decomposing the referring expressions. The "Q" denotes the input referring content and the "A" denotes the answers of LLM. In most cases, we can obtain reliable target-related textual cues that do not contradict the original text input. Besides, we also notice that there are some cases in which the text is not sufficiently decomposed (e.g, the example (c)) or the LLM outputs redundant results (e.g, the example (f)), which hinders the model to benefit from progressive comprehension to some extent.Our work consists of multiple stages and utilizes L cls in TRIS [30] at each stage independently for response maps optimization. Here, we omit the index of stage n for clarity. L cls formulates the target localization problem as a classification process to differentiate between positive and negative text expressions. The key idea of L cls loss function is to contrast image-text pairs such that correlated image-text pairs have high similarity scores and uncorrelated image-text pairs have low similarity scores. While the referring text expressions for an image are used as positive expressions, the referring text expressions from other images can be used as negative expressions for this image. Thus, given a batch (i.e., B) of image samples , each sample is mutually associated with one positive reference text (i.e., a text describing a specific object in the current image) and mutually exclusive with L negative reference texts (texts that are not related to the target object in the image). Note that the number of batches is equal to the sum of the positive samples and the negative samples (i.e., B = 1 + L).Speficially, in each training batch, B image-text pairs {I i , T i } B i=1 are sampled. Through the language and vision encoders, we can get referring embeddings Q ∈ R B×C and image embeddings V ∈ R B×H×W ×C . Then, we obtain the response maps R ∈ R B×B×H×W by applying cosine similarity calculation and normalization operation. After the pooling operation as done in TRIS, we further obtain the alignment score matrix y ∈ R B×B . According to the L cls , for i th image in the batch, there is a prediction score y[i, :], where y[i, i] predicted by the corresponding text deserves a higher value (i.e, the positive one) and the others deserve lower values (L negative ones). Then classification loss for the i th image from the batch can be formulated as cross-entropy loss:L cls,i = - 1 B B j=1 1 i=j log 1 1 + e -y[i,j] + (1 -1 i=j ) log e -y[i,j] 1 + e -y[i,j] ,and the classification loss for the batch can be formulated as:L cls = 1 B B i=1 L cls,iThe i denotes the index for the visual image and the j denotes the index for the referring text.In Sec. 3.2, we have mentioned that the conditional referring module (CRM) utilizes the decomposed textual cues to progressively modulate the referring embedding via a modulation block across N consecutive stages, and then produces the image-to-text response map by computing the patch-based similarity between visual and language embeddings. Specifically, the modulation block is implemented by a vision-to-text and a textto-text cross-attention mechanism in cascade for facilitating the interaction between cross-modal features. In Fig. 8 , we give an overview of the block design. For the block at each stage, We concatenate one targetrelated cue and the L negative text cues obtained from other images as the conditional text cues and then obtain the visionattended cue features by the vision-to-text attention. Then by learning the interaction between referring embedding and different textual cue embeddings, the block is expected to enhance the integration of discriminative cues.We implement our framework on PyTorch and train it for 15 epochs with a batch size of 36 (i.e., L + 1) on a RTX4090 GPU with 24GB of memory. The input images are resized to 320 × 320. We use ResNet-50 [11] as our backbone of image encoder, and utilize the pre-trained CLIP [43] model to initialize the image and text encoders. The down-sampling ratio of visual feature s = 32, the channels of vision feature C v = 2048, text features C l = 1024, and the unified hidden dimension C = 1024. The network is optimized using the AdamW optimizer [37] with a weight decay of 1e -2 and an initial learning rate of 5e -5 with polynomial learning rate decay. For the LLM, we utilize the open-source powerful language model Mistral 7B [16] for referring text decomposition. For the proposal generator, we set the number of extracted proposals P = 40 for each image.B More Quantitative Studies In Tab. 6 and Tab. 7, we conduct the ablation studies about the measurement criterion of alignment score. The results demonstrate that the maximum value of the response map in each proposal better represents the alignment level of region-wise cross-modal alignment than the average value. To validate the effectiveness of the modeling the progressive comprehension, we also quantitatively compare the outputs of our method at different stages in Tab. 8. The results show that the localization results gradually improve with more discriminative cues integration, especially in the early stages.Comparison between IaD loss and others. In our IaD loss L IaD , we adopt a hard assignment for deriving the loss function as GroupViT [54] . The motivation is that we aims to get the pseudo mask prediction by the accurate peak value point (i.e., the hard assignment results) instead of relying on whole score distribution S(•) (e.g., S a , S d in Sec. 3.4). Thus utilizing the hard assignment to derive the IaD loss well matches our purpose, which helps rectify the ambiguous localization results. If we use the soft assignment (e.g., measuring KL divergence between S a and S d ), though the equivalent may be simpler, it not only does not match our purpose but also introduces more tricky components for optimization (e.g., extra distribution regularization is required). In order to verify the argument, we conduct a comparison on RefCOCOg(G) val dataset as Tab. 9. The L IaD even causes a slight decline, while the proposed loss L KL brings clear improvement on localization accuracy.There are essential differences between them. First, the calibration loss in TRIS [30] is used to suppress noisy background activation and thus help to re-calibrate the target response map. In contrast, in our method, we observe that, multiple referring texts corresponding to different instances in one image may locate the same instances (or we say overlapping), due to the lack of instance-level supervision in WRIS. As for the implementation, the calibration loss adopts the global CLIP score of image-text to implement a simple contrastive learning for revising the response map. Differently, we simultaneously infer the response maps of different referring texts from the same image, and obtain the instance-level localization results by choosing the mask proposal with max alignment score. To further verify the superiority of our loss, we conduct an ablation on the RefCOCO(val) dataset. We use the TRIS without calibration loss as the baseline and then separately introduced these two loss functions for comparison. Both ablations demonstrate that the IaD loss not only refines the response map (mIoU) but also significantly improves the localization accuracy (PointM).Progressive Comprehension for Localization. In Fig. 9 , we also give visualization of each stage's response map for qualitative analysis. The results show that our proposed CRM module can effectively integrate the target-related textual cues. For example, in the first row, the method produces ambiguous localization result at the first stage. After taking the cue "with gold necklace" into consideration, the attention is transferred to the target object at the second stage. Finally, after considering all the cues, the method produces less ambiguous and more accurate localization results. Qualitative Comparison with Other method. More qualitative comparisons of our method with other methods are shown in the Fig. 10 . For the example shown in the fifth row, the query is "a man in a arm striped sweater". The TRIS [30] mistakenly locates the left man as the target regions. In contrast, our PCNet optimizes the response map generation process by continuously modulating the referring embedding query conditioned on the target-related cues instead of a static referring embedding. As a result, our method can obtain more accurate localization result.In this work, we adopt the two representative pre-trained segmentors: FreeSOLO [52] and SAM [20] for proposal generation. Specifically, the FreeSOLO [52] is a fully unsupervised learning method that learns class-agnostic instance segmentation without any annotations. The SAM's training utilizes densely labeled data, but it does not include semantic supervision. This supervision does not contradict our weakly supervised RIS setting. More importantly, it offers a promising solution as an image segmentation foundation model and can be used for refining the coarse localization results from weakly-supervised methods into precise segmentation masks as done in the recent works [10, 63] . For the usage of SAM, We adopt the ViT-H backbone, the hyperparameter predicted iou threshold and stability score threshold are set to 0.7, and points per side is set to 8.In Fig. 11 and Fig. 12 , we give the generated mask proposals examples by FreeSOLO [52] and SAM [20] , respectively. We notice that there are often overlaps among the generated proposals. Thus, we refine the generated proposals by filtering out candidate proposals with small area (the threshold is set as 1000) and then selecting the ones with smaller intersection over union (the threshold is set as 0.8). Considering that the number of proposals generated by the segmentor may be different for different image inputs, in implementation, we maintain consistency by selecting the top 40 proposals with the largest area (P = 40). If fewer than 40, we simply complete it with an all-zero mask.While we do not foresee our method causing any direct negative societal impact, it may potentially be leveraged by malicious parties to create applications that could misuse the segmentation capabilities for unethical or illegal purposes. We urge the readers to limit the usage of this work to legal use cases.Q: "the guy wearing white sitting on the couch watching his friends play video games" Q: "a catcher rushing to make a play on the ball" Q: "a boy in black t -shirt and jeans bending by keeping his hands on knees" Q: "a smiling man with a small infant wearing a beige t -shirt and gray jeans" Justification: In Sec. 1, we make the claims to reflect the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide the information on the computer resources in the implementation details of Sec. 4.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper discusses societal impacts of the work in the Appendix Sec. E.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper properly cites the existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.† Joint corresponding authors.Note that all counts start at 0.Refer to the Appendix for more details.For a fair comparison, we remove its 2nd stage as it is used for enhancing pixel-wise mask accuracy. 
 
 Appendix:  


Decision: Initially, the paper received mixed reviews, but following the author's rebuttal, the reviewers have reached a positive consensus. Each reviewer noted that the authors effectively addressed all concerns raised in their initial feedback. Two reviewers now strongly support the paper with a score of 7. The paper's contributions are recognized for their clear motivation, strong presentation, originality, and extensive experimental support. The AC also agrees that the paper exceeds the acceptance standards.

Reviews: review: paper_summary: paper_summary: This paper proposes a Progressive Comprehension Network for weakly-supervised referring image segmentation, which mimics the human process of progressive understanding by breaking down sentences into segments and gradually narrowing down the target range. The main contributions include: A multi-stage Conditional Referring Module to progressively comprehend text cues. A Region-aware Shrinking Loss to constrain the target region to gradually shrink. An Instance-aware Disambiguation Loss to eliminate overlap between different instances.
strength_weakness: strength_weakness: 1. The concept of mimicking the human process of understanding concepts from coarse to fine-grained, along with multi-stage refinement for different parts of sentences, has been explored in prior REC research [1]. This approach involves parsing sentences into multiple parts and conducting multi-stage refinement. Applying REC's ideas [1] to WRIS can also be considered a contribution, but it should be thoroughly discussed in comparison to [1], highlighting any differences, and cited appropriately.
2. There is a lack of discussion and citation of recent WRIS work, such as [2], in related work and the main text. It is necessary to discuss and cite the latest WRIS work in both the related work section and the main table.

3. Section 3.4 is not entirely understandable. I suggest the authors re-describe this section to make it clearer and easier to understand.
  
4. The top of Fig.2 is not clear enough, especially the subscripts a and d, I suggest the author re-design this part.


 [1] Yang S, Li G, Yu Y. Dynamic graph attention for referring expression comprehension[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 4644-4653.

 [2] Dai Q, Yang S. Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13711-13722.
questions: questions: 1. Why is the range of Equation 6 set to [0,1]? Can the combination of other proposals (denoted as 𝑚_𝑏 ) be simply approximated by the complement of the foreground, since the proposals segmented by SAM nearly cover the entire image? In this case, if IoU(𝑅_𝑛,𝑚_𝑓) < IoU(𝑅_𝑛,𝑚_𝑏), wouldn't Equation 6 be greater than 1?
2. Does 𝑚_𝑓 change at different stages in Equation 7, for instance, when different proposals are selected at different stages? What issues could this cause? In Equation 7, are 𝑛 and 𝑛+1 reversed? According to Section 3.3, the ambiguity score should be lower at later stages.
3. Section 3.1 mentions that there are five phrases for each sentence, and Section 3.2 states that each stage uses one text cue, so there should be five stages. Why do the implementation details mention only three stages?
4. In Section 3.4, how do you sample extra 𝑁_𝑑 texts? Are they randomly sampled from multiple text descriptions corresponding to the same image? How do you ensure that the sampled text descriptions correspond to different target referents? Because in refcoco series dataset, one referent may have multiple text expressions.
5. In the bottom part of Tab.1, are the numbers all obtained by using a single peak point as the prompt for SAM? Did you try using the response map as the prompt for SAM?
limitations: limitations: Yes.
paper_summary: paper_summary: Inspired by human's step-by-step cognitive process for localizing a target object in an image, the paper proposes Progressive Comprehension Network (PCNet) for the task of weakly-supervised referring image segmentation (WRIS) where text is the only supervision signal. PCNet first decomposes a long, complex text description into multiple target-related cues using a large language model (LLM), and the specialized designed module, Conditional Referring Module (CRM), progressively refines the text-to-image response map using the generated text cues in multiple stages. Region-aware Shrinking (RaS) is introduced for constraining the latter stage to have a more compact, target-related response map, and Instance-aware Disambiguation (IaD) loss is proposed for differentiating between the target object and the other objects in the input image. The experiment results show the superiority of the proposed method.
strength_weakness: strength_weakness: - The proposed method uses the contrastive loss of TRIS, but not generates independent response maps from a positive cue and L negative cues. In Equation (3) and (4), it integrates all the text cue features and generates a single response map R_n that contains information from both positive and negative cues. This does not make sense. How did the authors differentiate between positive and negative cues when computing the classification loss L_Cls? (This is why I rated "poor" for presentation).
- IaD, the loss to differentiate between the target object and other objects in the same image is similar to the calibration loss used in TRIS; the idea is the same while the calibration loss uses CLIP image-text similarity for differentiation. The authors should have discussed about this loss in the paper, and also done an ablation study comparing the two losses.
- The PCNet uses the mask proposals generated by a pre-defined segmentation mask generator such as FreeSOLO and SAM. We cannot ignore the possibility of knowledge being distilled from the mask generator into PCNet, especially through RaS and IaD losses, main contributions of the paper. More specifically, we cannot be sure that the performance gains from using RaS and IaD do not come from the mask generator's knowledge.
questions: questions: Please rebut the above weaknesses. I also have minor questions:
- In Figure 2, the arrow comes from Q_{n+1} to Cls. How is Q_{n+t} used for computing the classification loss?
- How did the authors sample L negative textual cues? one cue from each of the examples in the same mini-batch?
- I am curious about the rationale behind Equations (2) and (3). Why did the authors use residual connection only after the mlp layer, not also after the self-attention layer like in Transformer?
- Does the order in which text cues are used affect the performance of the model? E.g., is there a performance difference between feeding into the CRM module in the order of q_0, q_1, q_2 and in the order of q_0, q_2, q_1?
limitations: limitations: The paper addressed the limitations of the work.
paper_summary: paper_summary: This paper proposes the Progressive Comprehension Network (PCNet) for the WRIS task. this model achieves visual localization by progressively incorporating target-related textual cues for visual-linguistic alignment. Although experimental results have demonstrated the effectiveness of this paper, several aspects of the paper lack clarity.
strength_weakness: strength_weakness: 1.	The Conditional Referring Module (CRM) is implemented through multiple cross-attentions, which is a relatively common approach, lacking novelty. 
2.	What is the general idea of classification loss mentioned in the paper? Please explain the fundamental research insights of this work.
3.	In the Region-aware Shrinking (RaS) loss, what does the ⊙ symbol represent in Equation 5? Please clarify its meaning. 
4.	This article only conducted ablation experiments on the RefCOCOg dataset, so the ablation experiments are insufficient. It is recommended to validate on multiple datasets (such as RefCOCO+, RefCOCO) to comprehensively evaluate the effectiveness of our method.
5.	The order of the tables is somewhat confusing, with Table 9 preceding Table 6. It is recommended to reorder the tables to make them more in line with logical sequence or reading habits.
6.	Which datasets were the ablation experiments in Tables 7-9 conducted on? Please specify.
questions: questions: None
limitations: limitations: See the weakness
paper_summary: paper_summary: Aiming for the WRIS task, the work proposes a novel framework to leverage target-related textual cues from the description for progressively localizing the target object. The authors use a Large Language Model (LLM) to decompose the input text description into short phrases which are taken as target-related cues and fed into a Conditional Referring module. Besides, Region-aware Shrinking (RaS) loss and Instance-aware Disambiguation (IaD) loss are proposed to facilitate fine-grained cross-modal alignment.
strength_weakness: strength_weakness: 1. In table 1, only the results refined by SAM[1] are reported. Considering that FreeSOLO [2] is an unsupervised segmentation model, I am curious about the results refined by FreeSOLO. 

2.  This work introduces multiple stages for progressive localization. It may increase the training and inference time of the model.  The time comparison of model training and inference may be needed.

[1] Segment anything ICCV2023

[2] Freesolo: Learning to segment objects without annotations CVPR2022
questions: questions: 1. The mask proposals by SAM are often part of the whole instance.  It will affect the refined results. Does this work do anything special to alleviate the problem? The author can give more details about this issue.

2. Table 1 shows the results of the pseudo labels obtained at the wealy-supervised training stage. Can the authors give the results after pseudo labels based on supervised training?
limitations: limitations: NaN

Comments: title: Official Comment by Authors
Thanks for your response
Thanks for your response
comment: We thank Reviewer oLx4 for reviewing our work and for raising the review score. We really appreciate it. We will include the discussions in our revision.
Thank you for your detailed response, which addressed all my concerns. I decide to raise the score. Please include the previously mentioned discussions in the final revision.
Thanks very much for your positive feedback that we have addressed your most concerns and your willingness to raise the score. Here we give further explanations about IaD loss and Q4.

- IaD loss. Thanks for your careful comments!  In GroupViT, the Grouping Block groups similar semantic regions using the hard assignment strategy during the group tokens training and inferring process. In our IaD loss, we also adopt a similar hard assignment for deriving the loss function.  The reason is that we aims to get the pseudo mask prediction by the accurate peak value point (i.e., the hard assignment results) instead of relying on whole score distribution $S(\cdot)$ (e.g., $S_{a}$, $S_{d}$ in Section 3.4). Thus utilizing the hard assignment to derive the IaD loss well matches our purpose, which helps rectify the ambiguous localization results. If we use the soft assignment (e.g., measuring KL divergence between  $S_{a}$ and  $S_{d}$), though the equivalent may be simpler, it not only does not match our purpose but also introduces more tricky components for optimization (e.g., extra distribution regularization is required). In order to verify the argument, we conduct a comparison on RefCOCOg(G) val dataset as follows. The KL_Loss even causes a slight decline, while the IaD loss brings a clear improvement on localization.

  | Methods                                 | PointM | mIoU |
  | :-------------------------------------- | ------ | ---- |
  | $\mathcal{L}_{\texttt{CLs}}$            | 51.7   | 25.3 |
  | $\mathcal{L}_{\texttt{CLs}}$ + KL_loss  | 51.2   | 24.8 |
  | $\mathcal{L}_{\texttt{CLs}}$ + IaD_loss | 53.1   | 26.6 |

- Q4. Yes, you are right. `refer_id` is important information for the data_loader sampler and adopted in TRIS and SAG. Here, we further consider and utilize the prior (i.e., different texts corresponding to different instances within the same image should activate different regions) for localization optimization.

 If there are any additional clarifications needed, please do not hesitate to reach out.
Thank you for your detailed response, which addressed most of my concerns. I would like to further discuss the IaD loss and Q4. In GroupViT, a similar operation is used to backpropagate gradients, but they do this to achieve hard assignment during inference, while the gradients are computed in a soft form. However, in this paper, there is no need for hard assignments during inference. Is there a simpler equivalent form to replace the current IaD loss?

Regarding the answer to Q4, can I understand that you utilize the prior of different texts corresponding to different instances within the same image using **refer_id** ? Is it used in TRIS?

I will raise my score if you can provide an explanation. Thank you very much!
Thanks for your response and your willingness to consider raising the score. We genuinely appreciate your time and the valuable feedback. We are pleased that our rebuttal has addressed your concerns. However, we notice that your current rating still suggests that our work  is still not qualified to be accepted. Considering that the discussion period is ending soon, if there are any remaining issues or concerns, we would greatly appreciate the opportunity to discuss them further with you.
Thanks very much for your positive feedback! We deeply appreciate that you decided to increase the score. We'll include the explanations and more convincing results in our revision according to your careful comments.
Thanks for the rebuttal. My concerns are addressed, therefore I increase my score.
Thanks very much for your positive feedback that we have addressed your concerns through our response. Your decision to upgrade our score is deeply appreciated. Please let us know if you have further questions. We'll also include more explanations and new results in our revision as you suggested.
Thank the authors for the detailed response. The original formulation of the classification loss was quite confusing, but now it makes sense. Also, the comparison with the baseline + IoU loss convinces me of the contribution of the RaS loss. The authors also addressed my other concerns. It depends on further discussion with other reviewers, but I am leaning towards raising my rating. Please add the conducted comparison experiments with the calibration loss in the final draft.
Thanks very much for your positive feedback and further suggestions! We'll include more explanations and convincing results in our revision as you suggested.
I would like to thank the authors for their detailed responses. All my concerns were thoroughly addressed. Overall, this paper presents a reasonable and well-explained approach to mimicking the progressive process of understanding language instructions. By leveraging visual localization and segmentation tools (e.g., SAM), the method achieves superior language comprehension and target localization, supported by solid experiments and extensive visualization analysis. This work provides clear insights and significant contributions to the field. Therefore, I will maintain my original rating. I also recommend that the authors include suitable explanations and new results in the revision.