# Text-Guided Refinement for Referring Image Segmentation
## 3 Method
### Text-Guided Gating Mechanism  
For each referring expression, a text feature \(R\), where \(R\in\mathbb{R}^{C_{r}}\), is generated by the text feature encoder. Let \(F_{i}\) denote the input feature for the text-guided gated fusion mechanism,  
Figure 3: (**a**) The architecture of text-guided fusion. \(\oplus\) denotes element-wise sum. \(\bigcirc\) denotes concatenation. \(\otimes\) denotes matrix multiplication. \(C\), \(H\), and \(W\) are the channel number, height and width of feature maps, respectively. (**b**) The architecture of the bottleneck. where \(i\in 1,2,\ldots,N\), and \(N\) represents the number of input features. A spatial attention weight is learned between the text feature and the multi-modal feature \(F_{i}\) to quantify the importance of each spatial location in the feature based on its relevance to the text. The spatial attention weight is defined as follows:  
\[a_{i}=(RW_{R})(F_{i}W_{F})^{T} \tag{3}\]  
where \(W_{R}\in\mathbb{R}^{C_{r}\times C_{h}}\) and \(W_{F}\in\mathbb{R}^{C_{m}\times C_{h}}\) are learnable parameters, and \(a_{i}\in\mathbb{R}^{W\times H}\) represents the spatial attention weights. These weights are used to apply a weighted sum of the multi-modal feature \(F_{i}\), generating a feature that incorporates global information and enhances the regions of the multi-modal feature that are relevant to the referring description. Next, the global feature is concatenated with the text feature and passed through a fully connected layer to obtain \(G_{i}\in\mathbb{R}^{C_{m}}\). This feature is used to select the information from other scales that will be fused, as represented by the following equation:  
\[Y_{i}^{j}=(F_{j}W_{y})\sigma(G_{i}W_{\bar{g}}) \tag{4}\]  
where \(W_{y}\in\mathbb{R}^{C_{m}\times C_{y}}\) and \(W_{\bar{g}}\in\mathbb{R}^{C_{m}\times C_{\bar{g}}}\) are learnable parameters, and \(\sigma\) denotes the softmax function. \(F_{j}\) represents the feature from another scale that will be fused with \(F_{i}\), and \(Y_{i}^{j}\) denotes the information selected from \(F_{j}\) for fusion with \(F_{i}\). This procedure is repeated to obtain the information for fusion from other multi-modal features in the TG structure. Then, the information from other scales is aggregated into \(F_{i}\) as shown in Equation (5), yielding the preliminary fused feature \(F_{i}^{*}\) that contains a combination of multi-modal information:  
\[F_{i}^{*}=F_{i}W_{i}+\sum_{j=1}^{N-1}Y_{i}^{j} \tag{5}\]  
where \(W_{i}\in\mathbb{R}^{C_{m}\times C_{y}}\) are learnable parameters, and \(F_{i}^{*}\in\mathbb{R}^{W\times H\times C_{y}}\) represents the fused feature of scale \(i\), enriched with multi-modal information guided by the referring expression. This process is shown in Figure 3a. The features \(F_{i}^{*}\) obtained from different scales are fed into the gating structure. Similar to the approach in [12; 21], a gating function is used for further multi-scale feature fusion. Specifically, for each input feature \(F_{i}^{*}\), a memory gate \(m_{i}\) and a reset gate \(r_{i}\) are generated, where \(r_{i},m_{i}\in\mathbb{R}^{W\times H}\). These gate functions are similar to the gates in an LSTM, but for each input, the gate functions are computed independently, without any dependence on the order of inputs. Each input feature \(F_{i}^{*}\) is associated with a context controller \(C_{i}\), which regulates the information flow from other input features \(F_{j}^{*}\) to \(F_{i}^{*}\). This process is expressed as follows:  
\[C_{i}=(1-m_{i})\odot F_{i}^{*}+\sum_{j}^{N-1}\gamma_{j}m_{j}\odot F_{j}^{*} \tag{6}\]  
The fused feature \(Fusion_{i}\) is then computed as follows:  
\[Fusion_{i}=r_{i}\odot\tanh(C_{i})+(1-r_{i})\odot F_{i}^{*} \tag{7}\]  
where \(\odot\) denotes the Hadamard product, and \(\gamma_{j}\) is a learnable parameter that adjusts the proportion of information retained by the memory gate for each input feature. This parameter controls the flow of features from different scales (\(j\)) into the current scale (\(i\)).