Adaptive Selection based Referring Image Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-19 07:47:54

Abstract: Referring image segmentation (RIS) aims to segment a particular region based on a specific expression. Existing one-stage methods have explored various fusion strategies, yet they encounter two significant issues. Primarily, most methods rely on manually selected visual features from the visual encoder layers, lacking the flexibility to selectively focus on language-preferred visual features. Moreover, the direct fusion of word-level features into coarse aligned features disrupts the established vision-language alignment, resulting in suboptimal performance. In this paper, we introduce an innovative framework for RIS that seeks to overcome these challenges with adaptive alignment of vision and language features, termed the Adaptive Selection with Dual Alignment (ASDA). ASDA innovates in two aspects. Firstly, we design an Adaptive Feature Selection and Fusion (AFSF) module to dynamically select visual features focusing on different regions related to various descriptions. AFSF is equipped with scale-wise feature aggregator to provide hierarchically coarse features that preserve crucial low-level details and provide robust features for successor dual alignment.  Secondly, a Word Guided Dual-Branch Aligner (WGDA) is leveraged to integrate coarse features with linguistic cues by word-guided attention, which effectively addresses the common issue of vision-language misalignment by ensuring that linguistic descriptors directly interact with masks prediction. This guides the model to focus on relevant image regions and make robust prediction. Extensive experimental results demonstrate that our ASDA framework surpasses state-of-the-art methods on RefCOCO, RefCOCO+ and G-Ref benchmark. The improvement not only underscores the superiority of ASDA in capturing fine-grained visual details but also its robustness and adaptability to diverse descriptions.

Main paper: 
 related work: 
 Referring Image Segmentation (RIS) is designed to localize objects within images guided by natural language descriptions. The early approach [15] utilized a fusion technique combining linguistic and visual elements through concatenation. Subsequent efforts [4, 17, 21, 22, 29, 38] have harnessed sentence-level textual features from the descriptive phrases, whereas other studies [1, 9, 34, 40] have adopted word-level textual features for textual representation. Given that natural language intrinsically contains structured data [44, 57] that can be exploited to align with visual features, certain methodologies have decomposed expressions into various components [19, 52, 56, 58] or implemented a soft division approach using attention mechanisms [7, 10, 18, 45, 55, 62, 64] .Recent work has adopted more efficient structures for visionlanguage fusion. LAVT [59] utilizes the Swin Transformer [35] for visual tasks and incorporates modules for vision-language integration in the last four layers of the visual encoder. In contrast, ReSTR [25] and CRIS [51] start by separately encoding visual and linguistic inputs with a dual encoder, then merging these features either through a multi-modal transformer encoder or a cross-modal decoder. Inspired by the advancements in large language models [43, 48] , new studies approach RIS as an auto-regressive generation task, introducing capabilities for logical reasoning [26, 63] . Recently, VPD [65] explores using semantic data from diffusion models [14, 46] for RIS applications, while ReLA [33] and DMMI [16] extend RIS capabilities to handle multiple targets.Despite significant advances in RIS architecture, most methods face two main limitations. First, they depend on manually selected visual features from encoder layers, lacking flexibility to focus on language-preferred visual features for different objects. Additionally, most methods directly fuse word-level features with coarse visual features, leading to suboptimal performance. We introduce an innovative framework for RIS that seeks to overcome these challenges with adaptive selection and dual alignment of vision and language features.Vision-Language Model (VLM) is a type of deep learning model designed to simultaneously interpret visual and textual information. These models can be categorized into two workflows: singlestream and dual-stream. Single-stream models, such as [3, 5, 6, 36] , integrate vision and language embeddings using a unified selfattention encoder. In contrast, dual-stream models like CLIP [51] , ALIGN [20] , FILIP [61] , and GLIP [28] , employ separate encoders for each modality, aligning the outputs through a dot product. Other models such as ViLBERT [36] and LXMERT [47] utilize dual selfattention-based encoders to process within-modality interactions, while cross-attention mechanisms are used to handle interactions between modalities. As a milestone, CLIP [39] applies a contrastive learning approach across a vast dataset of image-text pairs, demonstrating significant transfer capabilities across over 30 classification datasets. Following this model, CRIS [51] uses a transformer decoder in conjunction with a CLIP model, adapting CLIP's text-toimage matching expertise to text-to-pixel applications. Given the outstanding performance of CLIP [39] , we follow this well-aligned vision-language model to implement our framework. 
method 3.1 overview: 
 The framework of our proposed Adaptive Selection with Dual Alignment (ASDA) is illustrated in Figure 2 . An image 𝐼 and referring expressions 𝑇 are fed to ViT-B based CLIP visual encoder (12 layers) and language encoder [39] Section 3.3) . Finally, the Coarse-to-Fine Segmentation Decoder (CFS) processes 𝐹 𝑐𝑜𝑎𝑟𝑠𝑒 through Local Visual Attention and segmentation decoder to generate response masks 𝑀 ′ . The Word-Guided Coefficient Generator (WCG) first combines the attention-enhanced feature 𝐹 𝐴 with word-level language feature 𝑓 𝑇 using Visual and Language Local Attention, generating fine feature 𝐹 𝑓 𝑖𝑛𝑒 . Following this, the Gated Coefficient Generation produces the coefficients 𝑓 𝑐𝑜𝑒 𝑓 corresponding to the masks 𝑀 ′ . The final mask output 𝑀 ∈ R 𝐻 ×𝑊 is then obtained by applying a weighted sum operation between the masks 𝑀 ′ and the coefficients 𝑓 𝑐𝑜𝑒 𝑓 (see Section 3.4).Given an image 𝐼 and referring expressions 𝑇 , we extract layer-wise visual features 𝐹 (𝑖 )𝑉 and text features [𝑓 𝑇 , 𝑓 𝐸 ] through ViT-B-based CLIP visual encoder and language encoder, respectively.Image encoder. Following the design of vision transformer ViT, the image 𝐼 ∈ R 𝐻 𝐼 ×𝑊 𝐼 ×3 is patched and projected to 𝐼 𝑃 ∈ R 𝐻 ×𝑊 ×𝐶 , where (𝐻,𝑊 ) = (𝐻 𝐼 /𝑃,𝑊 𝐼 /𝑃) and 𝑃 indicates the resolution of each image patch. Then, 𝐼 𝑃 is fed into ViT [8] which employs 𝑁 transformer layers. And the output of layer 𝑖 is defined as 𝐹 Text encoder. For the given natural language referring expression, we extract word-level language feature 𝑓 𝑇 ∈ R 𝑁 𝑇 ×𝐶 using a modified Transformer [49] architecture described in CLIP. Here, 𝑁 𝑇 represents the number of word-level language tokens. The expression sequence is enclosed with [SOS] and [EOS] tokens to indicate the start and end of sequences. The activations of the [EOS] token are considered as the sentence-level language feature 𝑓 𝐸 ∈ R 𝐶 for the entire natural language expression.As shown in Figure 2 , the Adaptive Feature Selection and Fusion (AFSF) module consists of two main components: Adaptive Feature Selection (AFS) and Scale-Wise Feature Aggregator (SFA). Specifically, we design an adaptive selection network in ASF which is used to selectively identify the most relevant low-level visual feature 𝐹 𝐿 and mid-level visual feature 𝐹 𝑀 . Then it generates corresponding multi-scale features of low-level visual feature and mid-level visual feature, denoted as F𝐿 and F 𝑀 . Subsequently, the SFA module establishes the relationship between global language feature 𝑓 𝐸 and the highest-level local visual features 𝐹 𝐻 . Additionally, it fuses multi-scale features of low-level visual feature F𝐿 and mid-level visual feature F𝑀 to enhance the visual information of the target object, generating coarse feature 𝐹 𝑐𝑜𝑎𝑟𝑠𝑒 .Adaptive Feature Selection (AFS). Based on our visual analysis in Figure 1 , the visual features in different layers of the visual encoder can capture different local highlights related to various objects. When selecting the low-level visual feature 𝐹 𝐿 , the input includes global visual feature of the low-level visual feature candidate layers 𝑓 𝑙 𝐶 , where 𝑙 ∈ {4, 5, 6}, and the sentence-level language feature 𝑓 𝐸 . We first compute the cosine similarity between 𝑓 𝐸 and the class token 𝑓 𝑙 𝐶 for each 𝑙 ∈ {4, 5, 6}:EQUATIONwhere 𝑠𝑐𝑜𝑟𝑒 𝑙 ∈ R 𝐶 , ⊗ means element-wise multiplication, Φ T is a linear layer that maps the global visual feature 𝑓 𝑙 𝐶 to the same dimension as 𝑓 𝐸 ∈ R 1×𝐶 . The score tokens 𝑠𝑐𝑜𝑟𝑒 𝑙 serve a pivotal role in determining the alignment and relevance between textual descriptions and visual representations at different levels. They measure the similarity of visual features to textual descriptions, illustrating the relevance degree of correspondence. We then design an adaptive selection network based on 𝑠𝑐𝑜𝑟𝑒 𝑙 to select the low-level visual feature:𝑠𝑐𝑜𝑟𝑒 𝐿 = Φ as ([𝑠𝑐𝑜𝑟𝑒 4 , 𝑠𝑜𝑐𝑟𝑒 5 , 𝑠𝑐𝑜𝑟𝑒 6 ]) 𝐿 = Φ argmax (𝑠𝑐𝑜𝑟𝑒 𝐿 ) 𝐹 𝐿 = 𝐹 (𝐿) 𝑉 (2)where [, ] denotes concatenation, Φ as means adaptive selection network which is a combination of a Linear layer and softmax, 𝑠𝑐𝑜𝑟𝑒 𝐿 ∈ R 3 represents the relevance scores of different feature layers processed through Φ as , 𝐿 is the index of the most relevant feature layer identified by Φ as using Φ argmax , and 𝐹 𝐿 is the feature of the ViT layer corresponding to the index 𝐿. Φ as here serves a critical function in identifying the most appropriate feature layer index from the concated score tokens. There are many possible network architectures for Φ as , but our experiments show that a simple combination of a linear layer and softmax yields the best results, as detailed in Table 5 . The extraction of mid-level visual features 𝐹 𝑀 follows the similar way applied for low-level features, employing our proposed Adaptive Feature Selection network Φ as to select the most related feature. This process is encapsulated as follows:𝑠𝑐𝑜𝑟𝑒 𝑚 = 𝑓 𝐸 • Φ T (𝑓 𝑚 𝐶 ), 𝑚 ∈ {7, 8, 9} 𝑀 = Φ argmax (Φ as ([𝑠𝑐𝑜𝑟𝑒 7 , 𝑠𝑐𝑜𝑟𝑒 8 , 𝑠𝑐𝑜𝑟𝑒 9 ])) 𝐹 𝑀 = 𝐹 (𝑀 ) 𝑉 (3)Thus, we obtain the low-level visual features 𝐹 𝐿 , mid-level visual features 𝐹 𝑀 . Here we consider the output of the last layer of ViT-B as the highest-level feature 𝐹 𝐻 , which captures more global information in the image. All these three features have the same dimension R 𝐻 ×𝑊 ×𝐶 . We then downsample the low-level and mid-level features. This process yields F 𝑀 ∈ R Scale-Wise Feature Aggregator (SFA). Inspired by the effectiveness of multi-scale features in the visual domain [11, 31, 67] , we have adopted multi-scale features to enhance feature alignment. Unlike traditional tasks in the visual domain such as object detection [11, 31, 67] and segmentation [27, 66] , maintaining the positional relationship between text and images when using multiscale features in RIS settings presents a challenge. To solve this problem, we introduce the Scale-Wise Feature Aggregator (SFA) to preserve the spatial relationship between text and images from CLIP while leveraging multi-scale features. SFA initially combines global language feature 𝑓 𝐸 with top-layer visual features 𝐹 𝐻 , using element-wise multiplication:EQUATIONwhere 𝑓 𝐻 ∈ R 𝐶 denotesEQUATIONwhere the down-sampled low-level visual feature F𝐿 is initially 2x up-sampled byΦ up to 𝐹 ′ 𝐿 ∈ R 𝐻 2 × 𝑊 2 ×32with channel reduction to 32. Concurrently, the down-sampled mid-level visual feature F 𝑀 is enhanced through Φ lateral to achieve 𝐹 ′ 𝑀 ∈ R 𝐻 ×𝑊 ×32 , with channel reduction to 32. Φ lateral here is a 1 × 1 Convolution-ReLU block. Based on our assumption that only basic information about the object location is needed, we boldly reduce the channel dimension to lower computational demands. The final aggregation step combines these enhanced features into 𝐹 coarse ∈ R 𝐻𝑊 ×𝐶 . This is achieved by employing a 3 × 3 convolution followed by a flattening operation via Φ agg . This process is designed to restore the channel dimensions and then flatten the output for subsequent stages, effectively maintaining a balance between preserving details and ensuring computational efficiency in our approach to integrating multi-scale features.As shown in Figure 4 , the coarse feature map of CRIS [51] become disrupted after interaction with word-level language features, often focusing on regions unrelated to textual description. The previous single-branch alignment method [25, 51, 54] utilizes a Transformer Encoder or Decoder architecture to fuse coarse features with wordlevel language features, as demonstrated in Figure 3 . To address this issue, we develop two distinct branches within the WGDA: Coarseto-Fine Segmentation Decoder (CFS) and Word Guided Coefficient Generator (WCG), which more effectively align word-level features with coarse feature 𝐹 𝑐𝑜𝑎𝑟𝑠𝑒 . Specifically, the CFS branch utilizes Local Visual Attention and Segmentation Decoder to discern relationships among visual tokens and generate masks 𝑀 ′ ∈ R 𝐻𝑊 ×𝑁 𝑐 focusing on different parts. The WCG branch comprises Visual and Language Local Attention and Gated Coefficient Generation, which find correspondences between word-level language features and local visual features to generate coefficients 𝑓 𝑐𝑜𝑒 𝑓 ∈ R 𝑁 𝑐 for different response masks 𝑀 ′ . And then we can get the final mask output 𝑀 ∈ R 𝐻 ×𝑊 by applying a weighted sum operation between masks 𝑀 ′ and coefficients 𝑓 𝑐𝑜𝑒 𝑓 . This design effectively preserves the alignment between global language and visual features while seamlessly incorporating word-level language features, avoiding the issues that come with single-branch.Coarse-to-Fine Segmentation Decoder (CFS). Within the CFS branch, we first leverage the coarse aligned feature 𝐹 𝑐𝑜𝑎𝑟𝑠𝑒 as inputs to Local Visual Attention module, resulting in the attentionenhanced local visual features 𝐹 𝐴 ∈ R 𝐻𝑊 ×𝐶 . For Local Visual Attention module with 𝑛 layers, the workflow of i-th layer is simplified as follows:𝐹 (𝑖 -1) 𝑎 = Φ MHSA (Φ LN (𝐹 𝑐𝑜𝑎𝑟𝑠𝑒 )) + 𝐹 𝑐𝑜𝑎𝑟𝑠𝑒 𝐹 (𝑖 ) 𝑎 = Φ MLP (𝐹 (𝑖 -1) 𝑎 ) + 𝐹 (𝑖 -1) 𝑎 . 𝑖 = 1, 2, ..., 𝑛 𝐹 𝐴 = 𝐹 (𝑛) 𝑎 (6)where 𝐹 (𝑖 -1) 𝑎 represents the gradual refined visual features, Φ MHSA indicates a multi-head self-attention layer and Φ LN means Layer Normalization. After 𝑛 layers of self-attention interaction, the Segmentation Decoder leverages the attention-enhanced visual features 𝐹 𝐴 to produce the response masks 𝑀 ′ , which is computed as EQUATIONwhere Φ up represents the upsampling operation, Φ conv 3×3 denotes the convolution operation with a 3 × 3 kernel, 𝐹 (𝑖 ) 𝐴 denotes the feature after the 𝑖-th convolution and upsampling operation, and Φ conv 1×1 indicates the final convolution operation with a 1 × 1 kernel. After the last convolution, the Φ sigmoid function is applied to 𝐹 (4) 𝐴 and then produce the response masks 𝑀 ′ ∈ R 𝐻𝑊 ×𝑁 𝑐 . Note that the hyperparameter n and 𝑁 𝑐 is discussed in the Table 3 and 4 in ablation study.Word-Guided Coefficient Generator (WCG). Within WCG branch, we integrate the refined visual features 𝐹 𝐴 and the wordlevel language features 𝑓 𝑇 through the Visual and Language Local Attention module. The multi-head cross-attention layer is adopted to propagate fine-grained semantic information into the evolved visual features. The calculation is as follows:EQUATION)where Φ MHCA denotes the multi-head cross-attention layer, and 𝐹 ′ 𝐾 is the intermediate features. The evolved multi-modal fine feature 𝐹 𝑓 𝑖𝑛𝑒 ∈ R 𝐻𝑊 ×𝐶 which captures the relationship between word-level language features and local visual features is utilized for generating the coefficient 𝑓 𝑐𝑜𝑒 𝑓 through Gated Coefficient Generation module. The calculation is as follows:EQUATIONwhere Φ coef comprises two stacked 3 × 3 convolution layers and one 1 × 1 convolution layer, followed by the Tanh activation function. The WCG branch effectively captures the essence of visualtextual interplay, producing normalized coefficients 𝑓 𝑐𝑜𝑒 𝑓 ∈ R 𝑁 𝑐 . Coefficients 𝑓 𝑐𝑜𝑒 𝑓 are then used to guide the segmentation output, ensuring a coherent integration of cross-modal insights.Table 1 : Comparisons with the state-of-the-art approaches on three benchmarks. We report the results of our method with various visual backbones. U: The UMD partition. G: The Google partition. "-" represents that the result is not provided. IoU is utilized as the metric. Finally, the segmentation mask 𝑀 is obtained by weighting and summing the coefficients 𝑓 𝑐𝑜𝑒 𝑓 ∈ R 𝑁 𝑐 and the response maskEQUATIONwhere 𝑀 ∈ R 𝐻 ×𝑊 represents the final output mask, ⊗ denotes the element-wise multiplication, and Φ reshape is a reshaping operation that transforms the multiplied result into the desired output dimensions. Rather than adopting the contrastive learning proposed in CRIS, we supervise mask prediction through a linear combination of focal loss [32] and dice loss [30] . 
experiments 4.1 datasets: 
 We conduct extensive experiments on three benchmark datasets. and [EOS] tokens. We use the Adam optimizer to train the network for 30 epochs with an initial learning rate of 5 × 10 -5 , and we decay the learning rate in the 18 𝑡ℎ , 25 𝑡ℎ epochs with a decay rate of 0.1. We train the model with a batch size of 28 on 2 RTX 3090 GPUs.Metrics. Following previous works, we adopt two metrics to verify the effectiveness: overall Intersection over Union (IoU) and Precision@𝑋 . IoU calculates the ratio of intersection to union regions between the predicted segmentation mask and the ground truth. Precision@𝑋 measures the percentage of test images that achieve an IoU score exceeding the threshold 𝑋 , with 𝑋 values of 0.5, 0.7, and 0.9.We evaluate our ASDA against state-of-the-art methods that utilize various visual and language backbones. Table 1 shows performance comparisons on three common splits of RefCOCO, RefCOCO+ and G-Ref. Compared to the recently SOTA ReLA [33] which uses Swin-B visual backbone [35] and BERT language backbone [24] , our ASDA improves IoU by 1.24%, 0.66% and 1.18% on val, testA and testB splits of RefCOCO, respectively. On RefCOCO+, ASDA shows improvements of 0.80%, 0.11%, and 0.18% respectively for val, testA, and testB splits. Furthermore, on G-Ref dataset, improvements are 0.73% for val (U), 0.48% for test (U), and 0.85% for val (G) splits. This demonstrates that our ASDA not only achieves a better understanding of the location and appearance information in RefCOCO but also adapts to the various forms of expressions in RefCOCO+ and G-Ref. Besides, the following two comparisons show the effectiveness of our ASDA from different perspectives: (1) CRIS [51] is the first method which uses CLIP-R101 in RIS task. Compared to CRIS, our ASDA demonstrates significant performance improvements, with average gains of 4.6%, 3.92%, and 5.13% on RefCOCO, RefCOCO+, and G-Ref datasets, respectively. This demonstrates Method val test A test B P@0.5 P@0.7 P@0.9 IoU P@0.5 P@0.7 P@0.9 IoU P@0.5 P@0.7 P@0. To verify the effectiveness of our proposed components, we conduct comprehensive ablation studies to investigate each component on the RefCOCO val, test A, test B dataset. The components studied include Adaptive Feature Selection (AFS), Scale-Wise Feature Aggregator (SFA), Coarse-to-Fine Segmentation Decoder (CFS), and Word-Guided Coefficient Generator (WCG). The main results of the ablation study are presented in Table 2 . Additionally, we have conducted ablation studies on the number of layers 𝑛 in the Local Visual Attention module, the number of channels 𝑁 𝐶 in 𝑓 𝑐𝑜𝑒 𝑓 ∈ R 𝑁 𝑐 and 𝑀 ′ ∈ R 𝐻𝑊 ×𝑁 𝑐 and the architecture of adaptive selection (AS) network, which are presented in Table 3 , 4 and 5 respectively. We limit the training of experiments in Table 3 , 4, 5 to only 10 epochs and present the result on val split of RefCOCO , which leads to some differences in results compared to those in Table 2 . Baseline and manual assignment. (a) The baseline method extracts the highest-level visual feature 𝐹 𝐻 from the last layer of the visual encoder and fuses it with the global textual feature 𝑓 𝐸 using element-wise multiplication. The fused feature is then processed by the segmentation decoder to produce the final mask. (b) We enhance the baseline by employing a manual assignment approach following CRIS [51] , manually selecting features from Layer 6 and Layer 9 as the low-level feature 𝐹 𝐿 and middle-level feature 𝐹 𝑀 respectively. These features 𝐹 𝐿 , 𝐹 𝑀 and 𝐹 𝐻 are then fused using the typical concatenation and projection method [51] , without the use of multi-scale transformations. This improves 2.68%, 2.85%, 3.45% IoU on RefCOCO val, test A, test B respectively over (a), which demonstrates the significance of utilizing features from intermediate layers which focus on local regions. Effect of Adaptive Feature Selection (AFS). (c) We replace the manual assignment (in Table 2 row 2) with our proposed Adaptive Feature Selection (AFS) module (in Table 2 row 3), which dynamically selects visual features using the global language feature 𝑓 𝐸 . In our setting, AFS will produce muti-scale visual features, here we just use the same fusion and decoder module as in (b). The 2.4%, 1.92% and 1.7% improvement of IoU on three splits compared to (b) shows our AFS module can adaptively choose features that 2 row 4 ). Incorporating (c) with the SFA module, the IoU improves by 0.61%, 0.46% and 0.39% on three splits. This highlights the importance of multi-scale visual features in accurately capturing detailed characteristics of objects described in text. Effect of Word Guided Dual-Branch Aligner (WGDA). We compare our proposed Word Guided Dual-Branch Aligner (WGDA) with previously proposed single-branch approaches [7, 17, 51] . As shown in Figure 3 , (d) a single-branch approach directly interacts 𝐹 𝑐𝑜𝑎𝑟𝑠𝑒 from the ASFA module with word-level language features 𝑓 𝑇 , while (f) our WGDA method uses dual-branch to let inguistic descriptors directly interact with masks predictions. The results indicate that our WGDA outperforms the previous single-branch method by 0.41%, 0.6%, and 0.74% on three respective splits (in Table 2 row 5, 6 ). This indicates that our proposed Word Guided Dual-Branch Aligner (WGDA) can not only progressively refine visual features, but also effectively employ word-level language Figure 4 : Visualization results of feature maps and final predicted mask in CRIS [51] and ASDA respectively. The coarse feature map represents the features without interaction with word-level language features, while the fine feature map results from the fusion and interaction between word-level features afterwards.features 𝑓 𝑇 to guide coarse feature 𝐹 𝑐𝑜𝑎𝑟𝑠𝑒 to focus more on wordlevel language features, highlighting the importance of precise vision-language alignment. Abalation about Adaptive selection (AS) network. In Table 5 , we experiment different architectures within the adaptive selection (AS) network. In this experiment, we use our full model, only replacing the AS network layer. Specifically, we investigate three structures in the Adaptive Feature Selection (AFS) module to compute relevance scores: Linear, Convolution (Conv), and Multilayer Perceptron (MLP) layers. Each structure includes a softmax layer that transforms the scores into a probability distribution. Our findings indicate that the simple Linear layer configuration outperformed both the MLP and Conv layers, as evidenced by improvements in IoU of 0.98% and 0.52% over the val split of RefCOCO. This demonstrates that the Linear layer's simplicity and efficiency contribute to better generalization and performance. Abalation about hyperparameter In Table 3 , We conduct experiments to determine the most appropriate number of layers for Local Visual Attention within the Coarse-to-Fine Segmentation Decoder (CFS). We find that optimal performance is achieved when 𝑛 = 2. In Equation 10, we obtain the final mask through weight and sum between response mask 𝑀 ′ ∈ R 𝐻𝑊 ×𝑁 𝑐 and coefficients 𝑓 𝑐𝑜𝑒 𝑓 ∈ R 𝑁 𝑐 . As shown in Table 4 , we conducted ablation studies on the hyperparameter 𝑁 𝐶 and found that the best performance is achieved when 𝑁 𝐶 = 32. As 𝑁 𝐶 exceeds the optimal value 32, performance declines due to overly complex model structure.In Figure 4 , we visualize the feature maps and the final predicted mask in CRIS [51] and our ASDA. Comparison of Feature Maps. Comparing the coarse feature map, which is the fusion results between visual features from visual encoder and sentence-level language feature 𝑓 𝐸 , both CRIS and ASDA roughly capture the objects' semantic information. However, after interacting with word-level language features, the fine feature map in CRIS struggles to capture text-relevant features, leading to incorrect predictions. In contrast, the fine feature map from our ASDA, obtained after the Word Guided Cross-attention module in the Word-Guided Coefficient Generator (WCG) branch, can capture local visual detail. As shown by the yellow box in the Figure 4 , our method's fine feature map can directly focus on visual features relevant to the text. Comparison of Predicted Mask. The final predicted mask clearly demonstrates the superiority of our method. As indicated by the red box in the figure, for expression (a), CRIS fails to finely segment the part of the woman obscured by a bottle, whereas our method produces a detailed mask. In scenario (b), among multiple easily confusable objects, CRIS struggles to identify the correct person, while ASDA accurately locates the target, as already detected by fine feature map . For (c), CRIS incorrectly identifies the object, whereas ASDA produces an accurate and refined mask. More visualization results are provided in the Appendix. 
conclusion: 
 In this paper, we introduce the Adaptive Selection with Dual Alignment (ASDA) framework for Referring Image Segmentation. Initially, our Adaptive Feature Selection and Fusion (AFSF) module dynamically selects visual features from vision encoder layers related to various descriptive texts. AFSF includes a scale-wise feature aggregator to provide hierarchically aggregated features that preserve crucial low-level details and robust features for dual alignment. Secondly, we utilize a Word Guided Dual-Branch Aligner (WGDA) that integrates visual features with linguistic cues through word-guided attention, effectively addressing vision-language misalignment by allowing linguistic descriptors to directly interact with mask predictions. This ensures focus on relevant image regions for robust predictions. Our extensive experiments show that ASDA outperforms state-of-the-art methods on the RefCOCO, RefCOCO+, and G-Ref benchmarks.  
: 
 The previous method utilizes features from fixed layers within the visual encoder regardless of the linguistic content, while our method adaptively selects the language-preferred features based on the specific language input. (b) Qualitative comparisons reveal that our ASDA framework outperforms previous state-of-the-art methods, achieving the best results across all splits on three datasets.Unlike standard semantic segmentation [10, 12, 13] , which categorizes image pixels based on a fixed set of labels, RIS needs to understand free-form language expression to locate the exact pixels of the referenced object. Hence, the primary challenge of this task involves achieving precise alignment between the relevant visual content and the descriptive text at the pixel level, which is crucial for accurately producing the necessary mask.Existing methods [15, 29, 40] often leverage external knowledge to facilitate learning, typically relying on separate vision and language encoder, such as the Swin visual encoder [35] paired with the BERT language encoder [24] , which inherently lacks multi-modal correspondence. Meanwhile, some studies [51, 54] have built on well-aligned model like CLIP [42] , leveraging the advantages of vision-language pretraining. Regardless of the backbone choice, these methods commonly use manually determined visual features from the visual encoder layers, subsequently investing significant effort into feature fusion and alignment [4, 7, 18, 19, 23, 29, 34, 45, 51, 62] . Although these methods have promising performance, they encounter inherent challenges. Primarily, most methods rely on manually selected visual features from the visual encoder layers. As illustrated in Figure 1 , when processing the same image with varying referring texts like "a sheep on the left" and "a cow on the right", these methods extract the same manually determined visual features. This manner lacks the flexibility to selectively focus on visual features that are more relevant to the specific language used. Moreover, the direct fusion between irrelevant visual features [7, 17, 51] with sentence-level language features introduces noise to subsequent word-level language features guided mask prediction, which results in disrupting the established vision-language alignments. As depicted in Figure 4 , by visualizing the coarse features and the fine features, we observe that the fine features becomes disrupted after the interaction between the coarse features and word-level language features, focusing on regions unrelated to the textual description.To this end, we propose an Adaptive Selection with Dual Alignment (ASDA) framework to enhance performance on RIS tasks, which mainly contains Adaptive Feature Selection and Fusion (AFSF) module and Word Guided Dual-Branch Aligner (WGDA) module. The Adaptive Feature Selection and Fusion (AFSF) module composes of Adaptive Feature Selection (AFS) module and Scale-Wise Feature Aggregator (SFA) module. Initially, AFS leverages sentence-level language features to select the most relevant features from the visual encoder layers. As shown in Figure 1 , our AFS module can adaptively choose features that are more relevant to object described in the referring expression. Subsequently, SFA merges multi-scale visual features from AFS with sentence-level language features, effectively integrating semantics with the visual attributes of various layers. This provides hierarchically aggregated features that preserve crucial low-level details and offer robust features for successor dual alignment. Furthermore, we design two distinct branches in WGDA: the Coarse-to-Fine Segmentation Decoder (CFS) and the Word Guided Coefficient Generator (WCG), to more effectively interact word-level features with coarse features. The CFS utilizes robust features from AFSF module to generate candidate masks. Meanwhile, the WCG module employs a word-level language feature guided attention mechanism to generate the coefficients to combinate these masks. Unlike previous methods [22, 51, 59 ] that directly merge word-level language features into coarse features, our WGDA enables linguistic descriptors to directly interact with mask prediction without compromising the already aligned features, which helps get finely aligned visual feature.In summary, the contribution of this work is fourfold:• We propose an Adaptive Selection with Dual Alignment (ASDA) framework to enhance performance on RIS task. • Our Adaptive Feature Selection and Fusion (AFSF) module dynamically selects the most relevant visual features based on language features, moving away from the conventional fixed-layer selection to a dynamic, text-responsive feature selection mechanism.• Our Word Guided Dual-Branch Aligner (WGDA) module improves alignment and robustness of word-level features with visual features through a dual-branch structure, enhancing the interaction between linguistic descriptors and mask predictions. • Our ASDA achieves state-of-the-art results on the RefCOCO, RefCOCO+, and RefCOCOg datasets. 
 
 Appendix:  


Decision: This is a borderline case. This paper introduces an adaptive vision-language features alignment method for referring image segmentation. The original reviews are one borderline accept and three borderline reject. The main concerns center around the complexity of the proposed method and unfair experimental settings. The rebuttal successfully addressed some of the concerns and one negative reviewer raises the final rating to borderline accept. The authors are encouraged to incorporate all the comments in the final version.

Reviews: review: paper_summary: paper_summary: The paper proposes a framework for the Referring image segmentation task. Specifically, they propose a method to adaptively select visual features from different layers, instead of using manually selected layers. 

The proposed Adaptive Selection with Dual Alignment (ASDA) framework increases performance on Referring image segmentation (RIS) tasks. ASDA contains the Adaptive Feature Selection and Fusion (AFSF) module and the Word Guided DualBranch Aligner (WGDA) module. The AFSF module is composed of Adaptive Feature Selection (AFS) and Scale-Wise Feature Aggregator (SFA) modules. The WGDA module is divided into two branches: the Coarse-to-Fine Segmentation Decoder (CFS) and the Word Guided Coefficient Generator (WCG).

+ The AFSF module dynamically chooses the most relevant visual features from language features, shifting from the traditional fixed-layer selection to a dynamic, text-responsive feature selection process.

+ The WGDA module enhances the alignment and robustness of word-level features with visual features using a dual-branch structure, which improves the interaction between linguistic descriptors and mask predictions.
strength_weakness: strength_weakness: The paper addresses the problem of effectively utilizing vision features from different layers of an encoder, which is a problem worth investigating. The writing is easy to follow.

The proposed method is described clearly, and the framework is illustrated in a comprehensive way. In the experiment, the authors use ViT-B as the visual encoder and Transformer as the language encoder and initialize them with CLIP. The framework achieves better results on all three datasets RefCOCO, RefCOCO+, and G-Ref. This shows that ASDA not only enhances the understanding of the location and appearance information in RefCOCO but also adapts effectively to the diverse expressions in RefCOCO+ and G-Ref.
limitations: limitations: While the idea makes sense, besides standard benchmarks, there are few theoretical arguments or experimental results supporting adaptively selecting the layers over using fixed layers. 

The implementation uses a different backbone than previous approaches, making it difficult to compare, especially when the improvement is a few percentage points.

Figure 2 is too detailed. Including the specific architecture of each block is unnecessary and hinders comprehension.
paper_summary: paper_summary: The paper proposes a adaptive selection based method for referring image segmentation. The method adaptively selects features from certain layers from the visual encoder, instead of only using the last layer of the backbone. The paper also propose a scale-wise feature aggregator that fuses the global language features and a Coarse-to-Fine Segmentation Decoder to generate the final mask.
strength_weakness: strength_weakness: The paper is well written and easy to follow.

The framework achieves sota performance on multiple datasets.
limitations: limitations: The architecture of the network looks quite complex, but we have seen many of the designs in previous works. e.g. the scale-wise is actually very similar with the FPN used in MCN, which also involves feature fusion in this step; coarse-to-fine segmentation decoder is quite like the mask decoder in VLT, and word-guided coefficient generator looks like some kind of weighted sum of candidate masks, which is essentially the same with the pipeline used in GRES.

The motivation is established but not well proved. Authors claim that the selected feature of AFS are “language-perferred”, but no evidence are given and more in-depth discussion and analysis are missing. Will certain words affect the selection? What are the difference between features from Layer 4 or Layer 6?

Visualizations of intermediate results are also missing. E.g. all candidate masks M’ and corresponding scores.

Eq.10. the element-wise multiplication of N_c and HW*N_c should also be HW*N_c. Did you mean matrix multiplication?

I am a little confusing with Figure 4: where are these features extracted and how to normalize?  

The network also looks very big, better provide the parameter size and FLOPs.


Summary: The main issue is that most of the parts of the network are similar with previous works, and the effectiveness of the proposed adaptive selection mechanism lack solid support.
paper_summary: paper_summary: This paper proposes a novel framework to solve the RIS task, using text features to guide segmentation for better visual language alignment.  Extensive comparative experiments demonstrate the effectiveness of ASDA and validation experiments demonstrate the effectiveness of the proposed module.
strength_weakness: strength_weakness: 1. The WGDA module is a novel design
2. The AFSF uses text features to guide visual feature seletction that can perform more visual-language alignment.
3. ASDF demonstrates strong performance
limitations: limitations: 1. Is θas really work？ It only has three neurons
2. It would be more convincing if you could provide the specific design of downsampling  and verify whether it is necessary.
3. It is difficult to understand the specific structure of WCG and CFS based on text descriptions only.
paper_summary: paper_summary: This paper proposed a method, ASDA, for refer segmentation. First, an Adaptive Feature Selection and Fusion (AFSF) module is designed to dynamically select visual features to generate coarse features. Then, a Word Guided Dual-Branch Aligner (WGDA) module is leveraged to integrate coarse features with linguistic cues by word-guided attention, producing a word-guided coefficient. Finally, the mask is generated by production of both branch.
strength_weakness: strength_weakness: 1. The proposed method demonstrates effectiveness on three employed datasets.
2. The idea of coarse-to-fine is quite novel.
limitations: limitations: 1. Why choose vision features on intermediate layers to generate coarse features? Why choose from layer [4,5,6] and [7,8,9] respectively? How the attention region shifts between intermediate layers should be more explained.
2. Why first downsample F_L and F_M, and then upsample them to generate the coarse features? 
3. The stage that refines the F_coarse is drawn to be on the text branch, but in fact, it refers to refining the coarse features of the vision branch, instead of injecting vision features into text features. Therefore, the proposed method actually only involves single-branch, that is injecting text features into vision features. It seems more like a dual-stage method, instead of a dual-branch method.
4. Lacks ablation study on layers chosen to generate coarse features ( in AFS ).
5. The concept of "stage" and "branch" should be unified, or it will result in confusion.