# Text-Guided Refinement for Referring Image Segmentation
## 3 Method
### Cascaded Pyramid Structure  
To effectively capture referring targets of various scales, we designed a cascaded pyramid network inspired by [39]. It consists of two main components: TGFusionNet and RefineNet (as shown in Figure 2). This architecture enables multi-modal features extracted at different encoder stages to be progressively aligned, fused, and refined during decoding. We first extract three levels of multi-modal features from the encoder, denoted as \(M_{3}\), \(M_{4}\), and \(M_{5}\), corresponding to the outputs of Res3, Res4, and Res5 in the backbone network. These features encode visual and linguistic information at increasing levels of semantic  
Figure 2: The overall architecture of the proposed approach. The encoder extracts features of the input image and description, subsequently performing an initial fusion to generate multi-modal feature. The decoder, represented by the Text-Guided Refinement Network, comprises two main components: TGFusionNet and RefineNet, which are enclosed in dashed boxes of different colors. TGFusionNet achieves cross-scale fusion of multi-modal features, while RefineNet further refines these fused features to enhance segmentation accuracy.  
abstraction and decreasing resolution. To prepare the features for fusion, a 1 \(\times\) 1 convolution is applied to each of \(M_{3}\), \(M_{4}\), and \(M_{5}\), yielding initial processed features \(M_{3}^{*}\), \(M_{4}^{*}\), and \(M_{5}^{*}\). The TGFusionNet consists of two pyramids, and its structure is similar to the feature upsampling schemes commonly used in semantic segmentation networks. It performs hierarchical feature fusion across these three scales. At each level, higher-level features are integrated into lower-level ones using a text-guided gating mechanism (TG), which selectively emphasizes spatial regions relevant to the referring expression. Specifically, \(M_{4}^{F}\), which integrates information from two scales, is obtained as follows:  
\[M_{4}^{F}=TG(M_{5}^{*},M_{4}^{*}) \tag{1}\]  
Here, TG represents the text-guided gating mechanism, and \(M_{i}^{F}\) represents the fused feature at scale \(i\) enriched by contextual and cross-scale information. Similarly, the feature \(M_{3}^{F}\), which integrates information from three scales, is computed as follows:  
\[M_{3}^{F}=TG(M_{5}^{*},M_{4}^{*},M_{3}^{*}) \tag{2}\]  
where \(M_{5}^{*}\), \(M_{4}^{*}\), \(M_{3}^{*}\in\mathbb{R}^{W\times H\times C_{F}}\), and the final fused feature \(M_{3}^{F}\in\mathbb{R}^{W\times H\times C_{F}}\).  
The output features \(M_{3}^{F}\), \(M_{4}^{F}\), and \(M_{5}^{*}\) are passed to RefineNet, which focuses on progressively restoring spatial detail and enhancing boundary precision. To improve the learning of scale-aware features and constrain TGFusionNet during training, each of these three features is also passed through two 1 \(\times\) 1 convolution for auxiliary segmentation prediction, and the corresponding loss is computed against the ground truth mask. In the refinement stage, RefineNet applies a variable number of bottleneck modules [40] to each input, followed by an additional text-guided fusion step to further align and integrate features across scales. The structure of bottleneck module is shown in Figure 3b. This stage is used to address challenging areas that may not be sufficiently resolved by TGFusionNet alone. Overall, the cascaded pyramid structure enables the coarse-to-fine alignment of visual-linguistic features, adaptive feature selection via text-guided gating, and progressive refinement for accurate mask prediction. The specific structure of the text-guided gating mechanism will be detailed in the following section.