Rethinking the Implicit Optimization Paradigm with Dual Alignments for Referring Remote Sensing Image Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 14:17:31

Abstract: Referring Remote Sensing Image Segmentation (RRSIS) is a challenging task that aims to identify specific regions in aerial images that are relevant to given textual conditions. Existing methods tend to adopt the paradigm of implicit optimization, utilizing a framework consisting of early cross-modal feature fusion and a fixed convolutional kernel-based predictor, neglecting the inherent inter-domain gap and conducting class-agnostic predictions. In this paper, we rethink the issues with the implicit optimization paradigm and address the RRSIS task from a dual-alignment perspective. Specifically, we prepend the dedicated Dual Alignment Network (DANet), including an explicit alignment strategy and a reliable agent alignment module. The explicit alignment strategy effectively reduces domain discrepancies by narrowing the inter-domain affinity distribution. Meanwhile, the reliable agent alignment module aims to enhance the predictor's multi-modality awareness and alleviate the impact of deceptive noise interference. Extensive experiments on two remote sensing datasets demonstrate the effectiveness of our proposed DANet in achieving superior segmentation performance without introducing additional learnable parameters compared to state-of-the-art methods.

Main paper: 
 related work 2.1 remote sensing image segmentation: 
 Remote sensing image segmentation is a crucial task in various scientific and engineering applications, facilitating the extraction of valuable information from satellite or aerial images [17, 32, 38] .Early researches on remote sensing image segmentation rely heavily on handcrafted features and simple thresholding methods [3, 26, 27, 36] , often struggle with complex scenes and lacked robustness to variations in illumination and terrain. With the advent of deep learning, there has been a paradigm shift towards data-driven approaches that learn hierarchical representations directly from the input data. CNN-based works [1, 2, 33] adopt well-known architectures like VGG [31] , ResNet [9] , and U-Net [30] , demonstrating superior performance in extracting spatial features from remote sensing images. Transformer-based methods [8, 35, 38] introduce the ability to model long-range dependencies, further enhancing the remote sensing scenes perception. However, traditional methods for remote sensing image segmentation often struggle to meet the specific foreground recognition requirements in practical applications. Therefore, referring remote sensing image segmentation (RRSIS) [23, 42] guided by textual cues has emerged. In specific, Yuan et al. [42] introduce a dataset for the RRSIS task and employed fine-grained textual information to guide visual encoding layer by layer. RMSIN [23] further incorporates a cross-scale interaction module to integrate multi-scale information from the encoders and used a fixed convolutional kernel to map the final layer visual features to predictions. Nevertheless, these existing methods mostly employ ambiguous text-visual feature fusion and utilize semantics-agnostic predictors for prediction, relying solely on implicit interaction to pray for ideal optimization. This implicit optimization paradigm ignores the inherent differences across domains and fails to mine valuable clues dealing with complex semantic scenarios. In this paper, we strive to address issues of the implicit optimization paradigm from an dual alignment perspective.Referring Image Segmentation (RIS), aiming to segment specific objects within an image based on natural language expressions [15, 21, 41] , has garnered significant attention in recent years. The task's goal is to understand and interpret linguistic descriptions, such as "the red phone" or "the plane on the right", and precisely locate and segment the referred objects in the image. Early RIS research [20, 40] focuses on extracting visual and linguistic features separately with Convolutional Neural Networks (CNNs) [18] and Long Short-Term Memory (LSTM) [10] networks, struggling to capture the relationships between language expressions and visual contents. The recent emergence of Transformer [34] architectures revolutionize RIS methodologies, offering remarkable fusion capabilities for multi-modality integration. For example, MDETR [16] showcases the effectiveness of simple concatenation of vision and language features followed by Transformer encoding and decoding for various vision-language tasks. Building upon this, VLT [5] pioneers the integration of Transformer architecture into referring segmentation, employing query generation modules to enrich language expressions with contextual image information. To facilitate cross-modal integration, LAVT [39] introduces language-aware attention mechanisms into image encoding processes, aiding early fusion of cross-modal features and improving segmentation accuracy. However, these existing referring image segmentation works primarily focus on the understanding of natural image domains, making it challenging to address complex terrains and specific contexts in aerial remotely sensed scenes. Thus, Yuan et al. [42] propose language-guided cross-scale enhancement to encourage the exploration of semantic clues in aerial images, while RMSIN [23] further introduces cross-scale skip connections to boost performance. Despite these efforts, these methods often fall into the paradigm of implicit optimization, neglecting the inter-domain gap and conducting semantic-agnostic prediction. We rethink this implicit optimization paradigm with dual alignments to better interpret referring remote sensing image scenes. 
method: 
 In this section, we first present the overview of the proposed DANet for RRSIS and the baseline model in Sec. 3.1. Then, we describe the details of the explicit affinity alignment strategy in Sec. 3.2 and the reliable agent alignment module in Sec. 3.3. Finally, in Sec. 3.4, the training and inference procedure are discussed.As shown in Fig. 2 , the input pair of referring remote sensing image segmentation contains an input image I ∈ R 𝐻 ×𝑊 ×3 and an input expression T ∈ R 𝐿 , where 𝐻 and 𝑊 refer to the height and width the input image I, 𝐿 denotes the number of words. For the feature extraction, we utilize the pre-trained language encoder (e.g., BERT) for encoding linguistic information as fine-grained embeddings L 𝑐 ∈ R 𝐿×𝐶 𝑡 and global textual representation L 𝑔 ∈ R 1×𝐶 , and take LAVT [39] as our baseline model, including Swin Transformer [24] as our backbone for vision feature extraction and PWAM (pixelword attention module, which is based on cross-attention) for vision-language fusion. During Stage 𝑖 in the encoder, the input feature F 𝑖 -1 is fused with text features L 𝑐 to output enriched vision feature F 𝑖 after downsampling. Please note that this part is not the focus of our design concerns, we use the same settings as LAVT for fair comparison.To mitigate the inter-domain gap caused by different pre-trained knowledge of vision and language encoders, the extracted hierarchical vision embeddings {F 𝑙 } 4 𝑙=2 are utilized by the explicit affinity alignment strategy to realize explicit alignments between vision and language domains. Through this operation of narrowing affinity distributions, the activation of global textual representation L 𝑔 in vision features gradually focuses on the ideal region. To empower the classifier's vision-aware capacity and further reduce deceptive background interference, L 𝑔 is regarded as a semantics-initialized prototype to align with selected reliable agents 𝐴 to acquire aligned predictor L𝑔 in the reliable agent alignment module. Finally, the top-level feature E from the backbone decoder is interacted with the evolved reliable predictor L𝑔 to generate the prediction Y.To tackle the problem of inter-domain gap caused by different pretrained knowledge in vision and language, we carefully design an explicit alignment strategy to narrow the domain discrepancies. Drawing inspiration from affinity learning, we explore explicitly aligning the vision and language domains by approximating affinity distributions. It can be noted that, in addition to leveraging the finegrained textual features L 𝑐 ∈ R 𝐿×𝐶 𝑡 , our design also strategically retains the [CLS] token L 𝑔 ∈ R 1×𝐶 from BERT outputs, serving as a macroscopic representation of global textual features. The [CLS] token in BERT serves a significant advantage: it encapsulates the semantic understanding of the entire input sequence, acting as a comprehensive sentence-level embedding that captures contextual information efficiently.Formally, during Stage 𝑖 in the backbone encoder, for the vision feature F 𝑖 ∈ R ℎ 𝑖 ×𝑤 𝑖 ×𝑐 𝑖 , we first flatten F 𝑖 as F𝑖 ∈ R ℎ 𝑖 𝑤 𝑖 ×𝑐 𝑖 , and the self-generated pixel-level affinity from F𝑖 is formulated as:EQUATIONwhere ⊤ refers to the transpose operation. Here, we denote the self-affinity map as A 𝐹 for clarity. The internal self-affinity of image features represents their internal consistency or self-similarity, each value in the self-affinity map reflects the correlation between the current observation area and other pixels within the same domain, indicating how well the features at different locations align with each other. To explicitly align the vision affinity with the language domain, we further utilize the global textual representation L 𝑔 to activate the corresponding semantic clues in vision feature F 𝑖 . F 𝑖 is mapped into the same dimension with L 𝑔 and produce the text-to-visual activation A 𝑡 𝑣 . To convert the multi-modal activation A 𝑡 𝑣 into hard labels, we conduct the transformation based on thresholding:EQUATIONwhere a(x,y) is the original pixel value from the activation map, and 𝛼 is the threshold for filtering positive samples. During implementation, we set 𝛼 as 0.80 by default, and as shown in Tab. 6, the variation of 𝛼 has insignificant impacts on performance, indicating the robustness of our design. Then, we aim to realize inter-domain alignment by narrowing the affinity distribution between vision and language clues. Specifically, we apply affinity constraints between these pseudo-labels and corresponding hierarchical visual features to narrow their distributions towards convergence, which can be formulated as:EQUATIONwhere ∥ • ∥ 𝐹 denotes the Frobenius norm, encouraging the distance between the two affinities to be similar and aligned. ⊙ denotes the element-wise multiplication, which signifies that we only focus on affinity constraints within the target, without enforcing consistency in the distribution of different categories in the background.One notable advantage of this explicit alignment strategy is that it operates within the text-visual encoding process without adding new learnable parameters, achieving gains in segmentation performance. This ensures generalization and scalability across multiple scenes, facilitating explicit visual-textual domain alignment in a principled and efficient manner.Existing approaches mostly use fixed semantics-agnostic predictors for dense prediction, struggling to mine valuable clues when facing complex semantic scenarios. In order to make the predictor semantic-aware and aligned to the linguistic domain, a preliminary scheme is to use global textual representations L 𝑔 as prototype, and directly align with the semantically-rich visual features X after the bottleneck to obtain a visual textually multi-aware classifier. However, such a naive correlation would inevitably fall into the trap of deceptive noise in remote sensing imagery, as shown in Fig. 1 (c ) left, destroying the purity of semantic features. How to achieve reliable visual-text alignment and avoid noise interference simultaneously is a problem worth solving. Intuitively, in a situation where it is difficult to distinguish between pixel classes, humans tend to decide by looking for a comparison between the current query location and a reliable class-aware region. Motivated by this intuition, assuming that given a trusted medium in remotely sensed imagery, we argue that the distribution of semantic and foreground pixels over that medium should be consistent, and vice versa. Thus we design a novel reliable agent alignment module (RAAM) to find dependable points as reliable medium (i.e., agent) to acquire more accurate correlations.Given the bottom-level image feature X after bottleneck (F 4 through feature enhancement) derived from the backbone encoder, the queries arise from the global textual representation L 𝑐 = {l 𝑛 } 𝐿 𝑛=1 , and keys and values arise from the input features X = [𝒇 1 ; 𝒇 2 ; ...; 𝒇 ℎ𝑤 ] (flattened X). Formally,EQUATIONwhere 𝑛 ∈ [1, . . . , 𝐿], 𝑚 ∈ 1, 2, ..., ℎ𝑤 and W 𝑄 ∈ R 𝐶 𝑡 ×𝐶 𝑘 , W 𝐾 ∈ R 𝐶 ×𝐶 𝑘 , W 𝑉 ∈ R 𝐶 ×𝐶 𝑣 are linear projections. Then, we can obtain the correlation between queries and keys with distance calculation as:EQUATIONwhere 𝑑𝑖𝑠 (•, •) denotes the distance metric. Direct pairwise alignment is unreliable due to deceptive backgrounds during remote sensing, especially with similar texture details, e.g., bikeways and sidewalks. The reliability for each pixel can be obtained via the weighted sum over all correlations as:EQUATIONwhere the top-𝐾 pixels are selected with the highest reliabilities (i.e., the largest correlations with semantics) to be agents A. The corresponding features can be denoted asF 𝐴 = {𝒇 𝐴 𝑘 } 𝐾 𝑘=1 .With agents that filter fine-grained semantic activations, we use them as mediators to achieve reliable alignment between visual features X and global textual representations L 𝑔 . We first map L 𝑔 to the query dimension (i.e., the same dimension with K 𝑚 ) asQ 𝑔 = L 𝑔 W 𝑄 𝑔, where W 𝑄 𝑔 ∈ R 𝐶 ×𝐶 𝑘 . Then, we calculate the agent-global semantics correlation 𝑐𝑜𝑟𝑟 𝑠 𝑛 and the agent-pixel correlation 𝑐𝑜𝑟𝑟 𝑝 𝑚 respectively as same as Eq.( 5):𝑐𝑜𝑟𝑟 𝑔 = softmax( Q 𝑔 (F 𝐴 W 𝐾 ) ⊤ √︁ 𝐶 𝑘 ), 𝑐𝑜𝑟𝑟 𝑝 𝑚 = softmax( (𝒇 𝑚 W 𝑄 )(F 𝐴 W 𝐾 ) ⊤ √︁ 𝐶 𝑘).(EQUATIONAnd then, we can obtain the reliable alignment between global semantics and pixels based on agents A as:EQUATIONwhich is used to acquire more accurate relations. Finally, the updated aligned classifier (i.e., clear vision-aware semantic representations) can be acquired by blending values with the reliable alignment 𝑎𝑙𝑖𝑔𝑛 𝑔&𝑝 𝑚 as:EQUATIONand following general transformer pipeline [34] , we equip the reliable classifier L𝑔 with self-attention and FFN at the output of the reliable agent alignment module. In this way, the global semantics L 𝑔 is modified in the RAAM and finally evolves into a reliable aligned predictor L𝑔 .With the final high-resolution features E ∈ R 𝐻 ×𝑊 ×𝐶 from the backbone upsampling decoder and the aligned predictor L𝑔 ∈ R 1×𝐶 , we can finally obtain the segmentation map Y as:EQUATION)For better training our network, we utilize the affinity constraint within the designed explicit alignment strategy and the conventionally used loss paradigm [23, 39, 42] , i.e., the binary cross-entropy loss for mask recognition. The final loss is formulated as:EQUATION)where Y is the ground-truth of the prediction Y, 𝜆 𝑎𝑓 denotes the coefficient of L 𝑎𝑓 𝑓 𝑖𝑛𝑖𝑡 𝑦 . Visual Enc. Text Enc. mIoU oIoU Pr@0.9 Pr@0.8 Pr@0.7 Pr@0.6 Pr@0.5 RRN [20] ResNet Visual Enc. Text Enc. mIoU oIoU Pr@0.9 Pr@0.8 Pr@0.7 Pr@0.6 Pr@0.5 RRN [20] ResNet  
experiments: 
 In this section, we will first introduce the datasets used in our work in Sec. 4.1. The implementation details are shown in Sec. 4.2. In Sec. 4.3, we illustrate the specific metric for better evaluation of our method. Then, we further analyze the main results including quantitative evaluations and qualitative results in Sec. 4.4 Finally, we ablate the effectiveness of our method in Sec. 4.5 for better demonstration of DANet's superiority.To demonstrate the effectiveness of our proposed model, we conduct extensive experiments on two referring remote sensing image segmentation benchmarks: RefSegRS [42] and RRSIS-D [23] .RefSegRS is a dataset containing 4,420 remote sensing images with language expressions and corresponding labels, divided with 2172 images allocated for training, 431 images for validation, and 1817 images for testing. The image resolution of this dataset is 512 × 512. RRSIS-D is a dataset with imaging resolution ranging from 0.5 to 30 meters and a resolution of 800 × 800. It consists of 17,402 aerial remote sensing images, with 1,740 images used for testing and the rest for training.We adopt Pytorch [28] and Detectron2 [37] to implement the proposed method. 4 NVIDIA GeForce RTX 3090 GPUs are used for training. We take the input image size as 480 × 480 following conventional settings [23, 42] . During the training stage, our model is trained with a batch size of 16, using the AdamW optimizer with an initial learning rate of 0.0005. We set the coefficient 𝜆 𝑎𝑓 of L 𝑎𝑓 𝑓 𝑖𝑛𝑖𝑡 𝑦 as 0.5, and the number of agents as 𝐾 = 32. We ablate the effects of these hyper-parameters in detail in our ablation studies (Sec. 4.5). Please refer to the supplementary material for more descriptions of implementation details and results.For a fair comparison, we adopt the same metric with previous works [23, 42] , including Mean Intersection over Union (mIoU), Object Intersection over Union (oIoU), and Precision at X pixels (Pr@X). The higher the values of these metrics, the better the performance. mIoU measures the average overlap between the predicted segmentation masks and the ground truth masks for each class. oIoU focuses on evaluating the segmentation accuracy at the object level rather than at the pixel level. It computes the IoU for each object instance and then calculates the average IoU across all objects. Pr@X evaluates the precision of object segmentation by considering the number of correctly predicted pixels within a certain distance threshold X from the ground truth masks.4.4.1 Quantitative Evaluations. Our method demonstrates superior performance in referring remote sensing image segmentation, outperforming state-of-the-art methods, as illustrated in Tab. 1 and 2. Evaluation on both RefSegRS [42] and RRSIS-D [23] datasets reveals compelling results. It can be observed that our method not only exhibits superior performance in terms of mIoU and oIoU, but also demonstrates substantial improvements when more precise and fine-grained segmentation is required (i.e., Pr@0.9 and Pr@0.8). Despite challenges posed by the implicit optimization paradigm in remotely sensed scenes, the explicit affinity alignment strategy and reliable agent alignment module in DANet enable accurate identification of foreground regions, mitigating deceptive environmental interference and the inter-domain gap. Activation maps shown in Fig. 5 further validate the superiority of our alignment design.As shown in Fig. 3 , DANet shows promising segmentation performance in diverse remotely sensed scenes. In specific, our method performs great in most scenarios for different targets. As shown in the third row in Fig. 3 , our approach allows for the accurate discrimination of confusable regions, e.g., as shown in the left of the 4 𝑡ℎ row, DANet isn't tricked by a similar "baseball field" to produce activation that shouldn't be there. And in the 2 𝑛𝑑 row, we can observe DANet achieves a high level of completeness in predicting foreground regions, consistent with our findings in Pr@X in Tab. 1 and Tab. 2. With explicit and reliable alignments, DANet can achieve more precise target localisation recognition. As shown in Fig. 5 , the activation maps with a reliable alignment strategy demonstrate stronger focusing capability, indicating that our method can fully leverage language guidance and alleviate domain discrepancies.We conduct comprehensive ablation studies on both datasets in terms of mIoU to verify the effectiveness of our modules. Impacts about Different Hierarchical Affinity Designs. As shown in Tab. 4, substantial performance improvements are evident on both datasets when employing multi-level features compared to utilizing only the bottom embedding 𝐹 4 . This underscores the effectiveness of our hierarchical affinity alignment design. The rationale behind this success is that vision-language alignments at different levels are well-suited for capturing targets at diverse scales. However, when applying the top-level feature 𝐹 1 at the start of EAAS, there exists a drop in performance. We deem that this setback is due to relatively weak ability to capture overall semantic information in 𝐹 1 , thus forcing it to align with semantics may disrupt the initial The design of our reliable agent alignment proves immensely beneficial to the model. This is particularly evident when it is integrated with explicit affinity alignments, as indicated in Tab. 3. In Tab. 5, we utilize different mechanisms to interact between vision features X and the textual representation L 𝑔 . Specific benefits brought by reliable agent alignments are demonstrated in Fig. 5 , as can be observed, activation without reliable alignment may be subject to deceptive background interference, resulting in erroneous activation or the loss of foreground integrity. Our designed reliable agent alignment module can suppress noise interference, leading to better aggregation and clearer foreground activation of various targets. Impacts about the Agent Selection Strategy in RAAM. In the reliable agent alignment module, the number of agents, denoted as 𝐾, determines how many reliable pixels are selected to establish the correlation between semantics and pixels. Ablating the number of reference points, as depicted in Fig. 4 (a), reveals that performance reaches its top when 𝐾 = 32, signifying that this number is sufficient for achieving the necessary correction. Too few reliable points may result in insufficient reference samples, leading to an unreliable medium between semantics and pixels, while too many agents may overly emphasize model discrimination and include irrelevant background information, ultimately disrupting the correlation distribution. The adaptive selection of agents in RAAM holds pivotal importance for pixel-level precision in challenging regions. 
conclusion: 
 In this paper, we rethink the implicit optimization paradigm to address the RRSIS task from a dual-alignment perspective. Specifically, we design a Dual Alignment Network (DANet) including an explicit alignment strategy and a reliable agent alignment module. The explicit alignment strategy effectively reduces domain discrepancies by narrowing the inter-domain affinity distribution, and the reliable agent alignment module enhances the predictor's multi-modality awareness and alleviates the impact of deceptive noise interference.Extensive experiments demonstrate our DANet's effectiveness. 
: 
 with the richer vocabulary and syntactic variations inherent in human natural language. To adapt to free-form text conditions, many works have have turned their attention to the referring remote sensing image segmentation (RRSIS) [23, 42] . Since the spatial and geographical differences conveyed from an aerial perspective differ from natural images, how to fully exploit the given text conditions to perform accurate and text-relevant visual region segmentation is thus extremely challenging.Currently, mainstream RRSIS methods draw inspirations from referring image segmentation (e.g., LAVT [39] ) credited to competitive performance, striving to align visual and linguistic branches. The core idea is, as illustrated in Fig. 1 (a) , RRSIS methods [23, 42] tend to resort to a paradigm of fusion-then-segmentation, including an implicit alignment between pre-trained textual (e.g., BERT) and visual (e.g., Swim-B) streams, and a fixed convolutional kernel for dense prediction. For example, Yuan et al. [42] directly integrate semantic features extracted by BERT into visual feature extraction, enabling semantic guidance for foreground focus, while RMSIN [23] further employs a cross-scale interaction strategy to replace skip connections and simply maps visual features to predictions with a convolutional kernel. Overall, these methods mostly employ ambiguous text-visual feature fusion and utilize semanticsagnostic predictors for pixel-level classification, relying solely on implicit interaction to pray for ideal optimization. These implicit optimization-based methods have made strides in referring remote sensing image segmentation indeed, but struggling to produce ideal results when dealing with complex semantic scenarios.Despite their promising results, after an in-depth analysis of implicit optimization that exists in the current fusion-then-segmentation paradigm, we find two key ingredients lacking in previous works.(1) Inter-domain Misalignment. During the visual-textual interaction process, the pre-trained knowledge of the language encoder originates from the natural language processing (NLP) domain, representing a discrete, structured data format, while the visual encoder focuses solely on the encoding and parsing of natural image domain, tending towards a continuous and high-dimensional representation. This notable gap between two domains, due to the inherent differences in training data, leads to a misalignment of input distributions in existing methods' implicit alignment paradigm (direct languagevisual interaction), resulting in ambiguous foreground activation, as shown in Fig. 1 (b) left. Designing an appropriate strategy to ensure the explicit alignment of visual and textual domain information is worthy of exploration. (2) Semantics-agnostic Prediction.Existing methods [23, 42] employ a CNN-based predictor for dense prediction (Fig. 1 (a) ), which remains fixed post-training and struggles to adapt to diverse visual-textual input pairs. Without explicit guidance from textual information, the fixed semantics-agnostic predictor fails to deeply understand objects and scenes in images, lacking generalisability across diverse linguistic contexts. Addressing how to endow the predictor with semantic perceptual capabilities to focus on specific targets is a desirable question.In this paper, we argue that pursuit alignments matter in RRSIS, which is intuitively sensible from the definition of the task itself. In this paper, we endeavor to mitigate the adverse effects of implicit optimization (misaligned encoding and semantics-agnostic segmentation), and aim to explore to achieve explicit alignments of visual-textual cues in the RRSIS task. Specifically, we design a coherent Dual Alignment Network (DANet) for referring remote sensing image segmentation, including an affinity-based explicit alignment strategy and an agent-based reliable alignment module. In the explicit affinity alignment strategy, we endeavor to explicitly alleviate the domain gap between vision and language. Intuitively, for the activation of textual information within visual features, pixels identified as foreground should exhibit a high correlation with each other. Drawing inspiration from this instinct, we harness insights from affinity learning [13] to seek perceptibility to spatial position and discrimination to surrounding pixels. We deem that there should be consistency between the affinity of text-to-visual activation and the internal affinity of visual features, as shown in Fig. 1 (b) right. Therefore, the explicit affinity alignment strategy is designed to utilize text-to-visual activation as pseudo-labels, applying affinity constraints between pseudo-labels and corresponding hierarchical visual features to narrow their distributions toward convergence, achieving explicit visual-textual domain alignment across multiple levels without the introduction of additional learnable parameters.In the reliable agent alignment module, our intention is to design a semantics-aware classifier with alignment with both domains. Since fixed classifiers cannot perceive semantic information post-training, a naive idea is to utilize pairwise alignment to update textual embeddings to become classifiers, as depicted in Fig. 1 (c ) left. However, in remote sensing scenarios, directly applying pixelsentence interaction inevitably increases the risk of unreliability due to the deceptive noise caused by long-distance imaging patterns in aerial imagery. In this case, segmentation networks are naturally equivocal for similar vision clues, leading to sub-optimal results. Hence, we aim to construct robust interaction with reliable matching to avoid uncertainty interference. Intuitively, as shown in Fig. 1 (c ) right, for the similarity distribution between a visual area and a group of pixels, the current distribution of similarity between the corresponding semantics and the same group should appear similar, and vice versa. Inspired by this intuition, we adaptively select reliable agents from vision features, avoiding deceptive visual interference caused by direct matching. In this way, we are able to leverage the purified textual knowledge to channel its attention towards discerning and prioritizing foreground regions. Overall, we manage to address problems of the implicit optimization in RRSIS from a dual-alignment perspective, leveraging a comprehensive framework that combines explicit affinity alignment and reliable agent alignment strategies. Through this integrated approach, we not only mitigate the challenges posed by domain discrepancies but also equip the predictor with semantic awareness, thereby enhancing its ability to discern and prioritize foreground regions accurately.In this work, our contributions can be summarized as follows:• We analyze the issues within the implicit optimization paradigm of referring remote sensing image segmentation, and propose to mitigate domain discrepancies from an explicit alignment perspective. • Specifically, we design a Dual-Alignment Network (DANet) including an explicit affinity alignment strategy to alleviate the inter-domain gap and a reliable agent alignment module to equip the predictor with semantic awareness. • Extensive experimental results on multiple challenging benchmarks demonstrate that our proposed method performs favorably against state-of-the-art referring remote sensing image segmentation methods. 
 
 Appendix:  


Decision: Referring Remote Sensing Image Segmentation (RRSIS) is a very novel and highly valuable research area. After one round of rebuttal from the authors, all reviewers have ultimately raised their scores to positive. Therefore, I believe this paper can be accepted as a poster.

Reviews: review: paper_summary: paper_summary: The task of Referring Remote Sensing Image Segmentation (RRSIS) involves identifying specific regions in aerial images that match given textual descriptions. Current methods often use implicit optimization, which includes early cross-modal feature fusion and a fixed convolutional kernel-based predictor. These methods neglect the inter-domain gap and perform class-agnostic predictions. In this manuscript, the authors propose a novel approach from a dual-alignment perspective to address these issues. The proposed DANet includes an explicit alignment strategy and a reliable agent alignment module. Extensive experiments on two remote sensing datasets show that DANet achieves superior segmentation performance compared with the SOTA methods.
strength_weakness: strength_weakness: 1. The proposed explicit alignment strategy could effectively reduce domain discrepancies and the reliable agent alignment module could enhance multi-modality awareness and mitigate noise interference.
2. Extensive experiments on the RefSegRS and RRSIS-D datasets demonstrate that the proposed methods could effectively boost the performance.
limitations: limitations: 1. The motivation shown in Figure 1 is not very clear, as it seems only show the difference between the proposed method and the existing method.
2. In addition to the current metrics (mIoU, oIoU, and Pr@X), inference efficiency, parameters, and computational complexity are also important and need to be further compared with comparison methods. The proposed reliable agent alignment module increases the size and the complexity of the model.
3. What are the advantages of the proposed DANet compared with the currently popular pre-training large multimodal models?
4. The performance of this work is limited and lacks comparison with the latest state-of-the-art methods.
5. The authors choose BERT as their text encoder. However, why not employ a more recent contextualized representation for the text encoder? A transformer-based representation might work better, such as GPT-2 and T5, or a multi-modal text encoder, such as CLIP text encoder.
6. Compared with the current large multimodal model, the value and significance of this work is not high enough. Authors should choose to proceed on a large multimodal model.
paper_summary: paper_summary: This paper discusses the two problems of the existing RRSIS method based on implicit optimization, namely, the misalignment between domains in the encoding stage and the semantic agnostic prediction in the decoding stage, and gives their own explicit alignment methods respectively, and combines them to form the dual alignment network proposed in this paper. The method proposed in this paper improves the segmentation effect of the model by aligning twice in the encoding and decoding stages, and verifies the accuracy of the model on two public datasets.
strength_weakness: strength_weakness: 1. By analyzing the problems existing in the existing implicit optimization strategy, a new dual alignment network is designed to solve the two problems of non-aligned encoding and unknown semantic segmentation from the perspective of explicit alignment.
2. A RAA module is proposed to select more accurate regions by comparing the distribution similarity between a group of pixels and their corresponding semantics, which effectively reduces the mis-segmentation caused by the deceptive noise generated by the long-distance imaging mode in the aerial image and improves the accuracy of segmentation.
3. Compared with the implicit optimization method, the explicit optimization strategy proposed in this paper achieves performance improvement without the need for additional learning parameters, and also improves the interpretability of the model.
4. A wide range of experimental results are provided, and the actual effect of the proposed method is verified on multiple benchmarks.
limitations: limitations: 1. Although the paper mentions that no additional parameters are added, the dual alignment process may still require more computational consumption, and the paper lacks relevant experimental tables.
2. The paper lacks verification of the robustness of the dual alignment strategy. The RRSIS-D data includes 15,000+ training data. If the amount of data is reduced, will the change in model performance decrease significantly?
3. The paper lacks discussion of the defects of the proposed method.
paper_summary: paper_summary: The authors rethink the issues existing in implicit optimization paradigms based on the traditional RRSIS methods, and propose solutions from the perspective of dual alignment. The authors designed two explicit alignment modules, namely explicit alignment strategy and reliable proxy alignment module. This method has shown significant advantages in both theory and practice, with high innovation and practical value. The effectiveness and robustness of the proposed method have been validated through extensive experiments conducted on two different RRSIS datasets. The experimental results show that the proposed DANet significantly improves segmentation performance without adding additional learnable parameters, outperforming existing state-of-the-art methods.
strength_weakness: strength_weakness: 1. The explicit alignment strategy proposed in the paper effectively reduces the difference between visual and text domains by reducing the cross domain affinity distribution. This method achieves consistency between visual features and text activation through affinity learning, avoiding the problem of fuzzy fusion in traditional methods. The explicit alignment strategy not only improves the model's cross domain alignment ability, but also enhances its processing ability for complex semantic scenes.

2. The reliable proxy alignment module aims to improve the multimodal perception ability of the predictor and reduce the interference of deceptive noise. By adaptively selecting reliable proxies in visual features, this module can better capture and utilize semantic information in remote perception scenes, avoiding uncertainty interference caused by direct matching. This module enhances the robustness of the model through reliable matching, making it perform better when dealing with complex visual cues.

3. The paper conducted extensive experiments on two different RRSIS datasets, and the experimental results proved the effectiveness and superiority of the proposed method. Compared with state-of-the-art methods, DANet performs well in multiple benchmark tests, significantly improving segmentation accuracy. These experimental results strongly support the theoretical assumptions and method design of the paper.
limitations: limitations: 1. There are some typos in the paper, such as "works have have turned" on page 1, line 67. It is recommended to carefully check and correct them.

2. There is ambiguity in Figure 1 (c). Is the Text Encoder in the Fixed Kernel-based Predictor? This may lead to confusion in readers' understanding of the method. It is recommended to provide detailed explanations or modifications to the illustrations to improve their clarity and accuracy.

3. The paper lacks a time complexity analysis of the proposed method. It is recommended to supplement time complexity analysis to comprehensively evaluate the efficiency and practical application value of the method.
paper_summary: paper_summary: This paper prepend the dedicated Dual Alignment Network (DANet) for RRSIS. Extensive experiments on two remote sensing datasets demonstrate the effectiveness of the proposed DANet in achieving superior segmentation performance without introducing additional learnable parameters compared to state-of-the-art methods.
strength_weakness: strength_weakness: 1.This paper proposed a noval framework for RRSIS and has clear idea.
2.The experimental results are good and the comparison methods are complete.
limitations: limitations: - 1.In Figure 3, almost all of them are large targets, and the questions asked are very simple, all about location and size. RRSIS-D has many cases of small targets, which are not visualized.
- 2.What are the application requirements of RRIS in remote sensing? For example, in the first line of Figure 3, there is only one bridge in the image, so what is the role of the text description. Even if text is not provided, the existing semantic segmentation model can extract the bridge very well.
- 3.The SAM model can also provide text descriptions as prompts. Can SAM also achieve the same effect? What is the significance of studying the specific RRIS model?